.. SPDX-FileCopyrightText: Samsung Electronics Co., Ltd
..
.. SPDX-License-Identifier: BSD-3-Clause

Changelog
=========

The repository is tagged using semantic versioning, e.g. ``v1.2.3``.

The ``next`` branch contains a preview of the upcoming release. The history on
``next`` is subject to change. Upon release, ``next`` is merged with ``main``
and  and tagged ``vX.Y.Z``.

Changes are described in this file in a section matching the version tag.

Known Issues
------------

See the file named ``ISSUES`` in the root of the repository.

v0.7.4
------

A lot of minor fixes, reduction of technical debt, and fixes in preperation for
Debian packaging. See the commits for details.

* Packaging, see subsection "packaging" in the Contributor section of the docs

* feat: Delayed identification, upon xnvme_dev_open() then a device was
  "identified" and its geometry "derived", this is dropped in favor of "adhoc" /
  "implicit" action when the data is requested.

* feat: The SPDK backend now supports iovec payloads

* feat: end-to-end data-protection is now available with helpers in the public
  API

* feat: Support for completion-notification via eventfd() by using helper
  function xnvme_queue_get_completion_fd()

v0.7.3
------

* Pseudo Commands

  - Encapsulation of functionality not NVMe-spec defined, yet common for
    drivers etc. encapsulated for backend-handling as "pseudo-commands" these
    are

    - xnvme_controller_reset()
    - xnvme_controller_get_registers()
    - xnvme_namespace_rescan()
    - xnvme_subsystem_reset()

  - Implemented in the SPDK and Linux backends
  - Usage via cli-tools is available

* Controller Handles

  - Having device-handles to NVMe-controllers is a necessity for e.g. reading
    controller-registers as provided via xnvme_controller_get_registers()

* Backend: SPDK

  - Support for the KV Command-Set over NVMe/TCP

v0.7.2
------

* API Refactoring

  - refactor(api): hoist inline-struct 'xnvme_opts.css'
  - refactor(api)!: hoist "xnvme_opts.oflags" and drop int-version
  - refactor(api): hoist and define xnvme_cmd_ctx_async
  - refactor(api): hoist and define xnvme_lba_range_attr

* Backends: libvfn/vfio

  - Fix use of the libvfn API (memory-mapping)
	- Bumping of the libvfn to libvfn 3.0.0-rc2, bumped to support
    eventfd-notification and fix errors with memory-mapping

* Backend: SPDK

  - feat(build/spdk): add libarchive dependency
  - feat(build/spdk): bump to v23.09

* Backends: Windows

  - Added support for SPDK in Windows via WPDK

Backend: linux / liburing

  - feat(be/liburing): turn on batching by default

* Feature: Pseudo-commands

	- Backend interface expanded with "pseudo-commands", this is to unify aspects
    which are not command-encapsulated; reading controller-registers, resetting
    subsystems etc.
	- api: add xnvme_subsystem_reset() / xnvme_controller_reset() /
    xnvme_namespace_rescan(), these construct "pseudo-commands" to instruct the
    backend to e.g. reset.

* Rust

  - xNVMe now has experimental Rust bindings via crate 'xnvme-sys'
  - feat(rust/xnvme-sys): add bindgen-emitted wrapper for libxnvme
  - feat(rust/xnvme_sys): add simplest example of call via bindings
  - feat(git/ignore): add 'rust/target' to ignore
  - feat(pcf,gha): add build of Rust crates


* Misc. Fixes

  - fix: format-strings (use PRI-macros instead of e.g. lu-specifiers)
  - fix(cli,enum): use xnvme_buf_virt_{alloc|free}() instead of malloc/free
  - fix: use zu format-specifier for size_t
  - fix(cli): change assignment for create_mode
  - fix(spec): adjust format-specifier for comp_temp
  - fix(be/libaio): remove comparison warning 
  - refactor(toolbox/xnvmec_generator): emit to subfolders
  - refactor(build/manpages)!: only install manpages for installed binaries
  - refactor(build/completions)!: only install completions for installed binaries
  - fix(meson/options): remove unused option shared_library
  - fix(meson/options): remove unused option enable-ysnp
  - fix(toolbox/pkgs): build libvfn as release

* Build

  - Cleanup helper targets

* cijoe

  - feat(cijoe): add workflow for provisioning via git

* Documentation

  - Added Tutorial for FDP
  - Added Overview of the xNVMe CI environment

v0.7.1
------

* API
  - Introduced memory-mapping API (libxnvme_mem.h), that is, when the user
    manages memory and thus does not want to use xnvme_buf_alloc() as doing so
    would introduce bounce-buffers.

* CLI
  - The passthru cli-interfaces now use --cdwXY arguments instead of
    "command-files", this alignes with nvme-cli and is arguably simpler

* Backends: Linux
  - Performance improvments in io_uring and io_uring_cmd backends due to the
    addition batching via delayed submission
  - Performance improvements to libaio due to the addition of the "ring-hack"

* Backends: Windows
  - Support added for writing via "ioring" on Windows, previously only batched
    reads were possible via this interface.

* Infrastructure; github-actions, docker-images, third-party etc. bump the
  reference toolchain platforms, that is
  - Debian is now Trixie, Bookworm and Bullseye.
  - Ubuntu is now Lunar, Jammy and Focal
  - Added Rocky Linux 9.2
  - Added Oracle Linux 9.
  - Dropped CentOS 7
  - Dropped CentOS Stream 8
  - Dropped openSUSE Leap 15.3 and 15.4, Tumbleweed remains

* Removal of a bunch of deprecated files and code. General *cleanup*.

v0.7.0
------

This release got a whole lot bigger than it ought to. Aiming for more
frequent releases in the future. A bunch of changes to API,
implementation, and even the license.

Regarding the license. For compatibility with GPL-licensed software.
That is, consumption of xNVMe from GPL-licences software, then xNVMe is
changed from APACHE to BSD-3-Clause. To better determine the license,
the project has aligned with the golden practice of REUSE-compliance
(https://reuse.software/).

* API
  * Refactored to provide a single "entry-point" header
  * Headers no longer include other headers (except for the entry-point)
  * Renamed the command-line api from xnvmec to xnvme_cli

* API / Command-Sets
  * Support for FDP and DSM
  * Support for Key-Value SSDs

* Backends
  * The POSIX backend is replaced by CBI -- Common Backend Implementation
  * Instead of "POSIX" then a handful of commonly useful implementations
    are provided, being "strictly" POSIX was an impractical restriction,
    such as lacking support for features which are provided on modern
    operating systems but not defined by POSIX.

* Backends: FreeBSD
  * Support for kqueue based aio

* Backends: SPDK
  * Now has options to control Admin and I/O command timeout
  * Removed patches no longer needed (available upstream)

* Backends: vfio
  * Removed as subproject and instead links "properly" with the
    system-provided libvfn

* tools
  * switched fio from "external" to "internal" / "upstream" fio engine

For additional details then please have a look at the commit-history,
the v0.7.0 related issue on GitHUB and PR.

v0.6.0
------

A handful of improvements for Windows, additional steps toward removing
third-party vendoring / bundling, addition of experimental tunable knobs for
io_uring/SQPOLL, and several improvements to testing and CI infra.

* API
  - Removed SLIST from API, although "sys/queue.h" are commonly available on
    Linux/FreeBSD, then they are not part of toolchain on Windows.

* Third-party
  - Bumped SPDK to v22.09, and with that removed mutliple out-of-tree patches
    for DPDK.

* CLI
  - The xNVMe command-line library (libxnvmec) and all the cli-tools using it
    are refactored to use common sets of command-line arguments. Along with
    this came a consistent set of CLI-arguments for admin/sync/async.

* Backends
  - ramdisk: The ramdisk got support for write-zeroes, iovec payloads and added
    to CI testing.
  - linux: support for a buffer-allocator using HUGEPAGES and tunable knobs for
    controlling the behavior of io_uring SQPOLL via environment variables.
  - windows: support for the experimental IORING Windows SPDK API and support
    for block devices (SCSI and SATA).
  - spdk: when controllers are re-used for device-handles, events are
    processed as a means to check whether the controller is still "alive"

* CI
  - scan-build now runs for each PR
  - basic tests are now running post-building testing using the RAMDISK
  - Testing of fabrics with TCP transport is now part of the setup

v0.5.0
------

A bit of expansion in the application of xNVMe with support for macOS, a
ramdisk backend, revival of Python language bindings and a refresh of the docs
on NVMe-over-Fabrics.

* API
  - Removed helpers for SGL
  - Add 'subnqn' to 'xnvme_ident', useful for NVMe-oF
  - Add 'hostnqn' to 'opts', useful for NVMe-oF
  - Add support directive-receive and write-with-directives
  - Spec adjusted for NVMe 2.0, still more work needed in this area

* Third-party
  - Bumped fio to v3.32
  - Bumped SPDK to v22.05
  - Removed liburing, now relies on on-system library instead of
    vendoring/bundling, documentation is updated to assist with library
    installation

* Tooling
  - Re-working testing using cijoe 0.9+, that is, switching to CIJOE/pytest for
    testing an CIJOE workflows for instrumentation
  - liburing is no longer bundled with xNVMe, that is, xNVMe now links with
    liburing as discovered on the system via pkg-config. This is done to avoid
    symbol collisions for applications linking or loading liburing and xNVMe.

* be:linux:async:libaio
  - When 'opts.poll_io' is set then poke() will return immmediatly and now wait
    for completions. This allows the use of trading CPU for more IOPS and lower
    per command latency.

* Additional user-space NVMe driver support via libvfn
  - Added 'be:vfio' providing another user-space driver via libvfn

* Preliminary support for macOS
  - Initial implementation using the "core" I/O mechanisms of sync-io, async
    emulation and the threadpool
  - Does enumeration of NVMe devices through the limited interface provided for
    user-space by the macOS kernel
  - Utilizes what is avaiable for admin-command submision

* Prelimiary support for a "ramdisk" device
  - be:ramdisk: added a backend mimicing the behavior of an NVMe NVM namespace
  - Intended as a test-vehicle providing a "device" without requiring anything
    but the consumption of main memory of the system
  - I/O is "stored" using main-memory

* Revival of the xNVMe Python language bindings
  - A re-introduction of the Python bindings, these are now generated and thus
    provide access to the entire xNVMe C API
  - They are added to the testing infrastructure ensuring that they are aligned
    with the library
  - In addition to simple ctypes bindings, then cython headers and bindings
    based on Cython are provided

* Documentation
  - Refreshed the NVMe-over-Fabrics tutorial
  - Expanded with descriptions on installing liburing
  - Expanded with a section for the WIP Python bindings

v0.4.0
------

This is a release with the sole purpose of changing the liburing subproject
from tracking 'master' to the next stable release that is liburing-2.2.

v0.3.0
------

This main feature of this release is the alignment of the ``io_uring_cmd``
implementation with the ``io_uring`` big-sqe/big-cqe approach to asynchronous
passthru of NVMe commands.

NOTE: the tracking of the liburing repository/subproject is changed from the
fixed tag ``liburing-2.1`` to the ``master`` branch. Thus, in case you
experience liburing related build-issues with this release, then it is most
likely due to changes on ``master``. As soon as ``liburing-2.2`` is released,
xNVMe will be released as well going back to stable tracking.  Thus, do not pin
your project to the xNVMe project tag for ``v0.3.0`` if you rely on the
``io_uring`` functionality.

* Asynchronous Passthru of NVMe Commands via ``io_uring``
  - There are no API changes to adjust to, the changes are encapsulated inside
    the implementation of ``be:linux:async:ucmd`` aka ``async=io_uring_cmd``.
  - The previous version of ``io_uring_cmd`` used indirect-commands, that is,
    the io_uring-sqe contained a pointer to the NVMe-command. This approach of
    passthrough via ``io_uring`` has been superseeded by the
    ``big-sqe/big-cqe`` approach with the NVMe-sqe embedded within the
    io_uring-sqe, and similar for the NVMe-cqe inside the io_uring-cqe.
  - This requires changes to how the ``io_uring`` is setup, this task is
    delegated to ``liburing`` and the subproject-wrap now tracks liburing
    ``master`` to do this.

* API
  - Fixed ``xnvme_enumerate()`` when ``NULL`` was passed as ``opts``, it now
    uses ``xnvme_default_opts()`` when no ``opts`` are given
  - Misc. fixes to docstrings missing descriptions

* cmd:
  - Fixed missing full-guard on full-guard in xnvme_cmd_passv()

* be:async:{emu,thrpool}: several fixes to command-processing
  - Fixed missing setup of completion errors
  - Fixed missing empty-guard in cmd_io{v}()

* fio IO engine
  - 3p:fio: bumped to v3.30
  - tools:fioe: fixed issue with iovec-payloads
  - tools:fioe: cleanup and alignment with upstream xNVMe fio IO engine
  - docs: removed deprecated information and re-written with usage examples

* tests:io_worker
  - Added a basic io_worker to verify the behavior of the
    submit-upon-completion

* tools:xdd
  - The ``xdd`` tool now provides an ``offset`` argument (in bytes), previously
    it started from 0

* Documentation
  - Re-introduced the ``tutorial`` section containing a guide to dynamically
    load xNVMe from C and Python
  - Added a Contributors section containing notes useful for first-time
    Contributors

v0.2.0
------

Main feature introduction is vectored I/O across a wider set of system
interfaces, that is via ioctl(), io_uring (ucmd) and preadv()/pwritev()
fallback.

* Support for vectored I/O via Linux: ioctl(), psync, and io_uring_cmd

* API
  - add xnvme_cmd_passv()
  - rename rename xnvme_queue_wait() to xnvme_queue_drain()

* be:io_uring_cmd:
  - Enabled NVME_IOCTL_IO64_CMD by default, when available for cmd_io()
  - Added support NVME_IOCTL_IO64_CMD_VEC over io_uring via cmd_iov()

* be:linux:nvme:
  - Enabled NVME_IOCTL_IO64_CMD by default, when available for cmd_io()
  - Added support NVME_IOCTL_IO64_CMD_VEC via cmd_iov()
  - Normalized error-handling for NVMe-ioctl interfaces, ioctl() as well as
    io_uring_cmd

* be:thrpool:
  - Added handling of cmd_iov(), providing a threadpool based fallback when
    io_uring_cmd is not available

* be:emu:
  - Added handling of cmd_iov(), providing a pseudo-async fallback when
    io_uring_cmd is not available

* Re-worked git-pre-commit using the 'pre-commit' framework
  - mk: added helpers invoking 'pre-commit', 'make format'/'make format-all'
  - mk: removed auto-setup of git-hooks
  - git: removed .githooks/pre-commit

* xNVMe fio io-engine
  - tools:fioe: use calloc instead of malloc
  - tools:fioe: changes according to fio coding conventions

* Library introspection
  - fix incorrect generation of third-party information
  - replace ``xnvme_3p`` with ``xnvme_libconf``
  - add all build-configs to ``xnvme_libconf``

* Command-line argumenter parser
  - xnvmec: fix missing setup of --direct

* CLI-fixes
  - zoned: fix description for identify namespace command

* Toolbox
  - mk: add script generating help-text on Makefile targets
  - meson: only do whole-archive in pkg-config when SPDK is enabled
  - scripts: replace astyle with clang-format
  - pcf: the pre-commit-framework is available for xNVMe

v0.1.0
------

Another infrastructure / fixes release.

* 3p:liburing
  - Bumped to 2.1
  - This breaks old distros: Debian Stretch and CentOS 7 but adds support for
    the latest Arch, Fedora, Tumbleweed, and Ubuntu

* docs
  - Added scripts and docs for: openSUSE, Fedora, CentOS Stream

v0.0.29
-------

Another infrastructure / fixes release.

* Re-worked the continous integration
  - Fixed the broken build of the "dockerized" source
  - Fixed build on FreeBSD
  - Added build and test of FreeBSD
  - Combined all workflows in a single workflow, this vastly improves how the
    CI is triggered and linked with artifacts and artifacts verified

* 3p:windows
  - Added definition for iovec, in preparation for iovec support

v0.0.28
-------

This and the previous release contain minimal library/logical changes as major
changes to the build-system and source organization is changed.

* Moved the libraries sources from 'src' to 'lib'

v0.0.27
-------

* Build-system migrated from CMake to meson
  The Makefile "frontend" to the build-system is still available, and
  instruments meson in the same manner it instrumented CMake. However, this is
  no longer intended for anything other than development. Meson is the way to
  go and the documentation thus describes how to use it rather than the
  make-helpers instrumenting meson.

* Reduced cpu-utilization on libaio and io_uring ``poke()`` implementations

v0.0.26
-------

Expanded platform support, updated experimental features, and extended
command-set-support for ZNS/ZRWA, along with a couple of fixes and third-party
updates.

* Third-party
  - fio, updated to 3.28
  - spdk, updated to v21.10

* Windows Support
  - xNVMe now builds on Windows, it uses the MinGW toolchain to be compatible
    with fio, however, xNVMe does also build with MSVC
  - Using IOCP for async I/O
  - Supports a limited number non-I/O commands via driver IOCTL mapping

* uring_cmd
  - Experimental interface updated for patch-set on top of 5.15 kernel

* Zoned Namespaces
  - Added support for Zone Random Write Area (ZRWA)

* Fixes
  - Linux Block Backend: fix and update sysfs processing
  - fio io-engine: Fix of xnvme_fioe_reset_wp() resetting one too many zones
  - Adjustments to CI and partly removed of deprecated 'schemes'

v0.0.25
-------

Major improvements to the usability of xNVMe and enchancements of the API
along with a couple of fixes.

* Encoding of runtime instrumentation, that is, selection of backend, async
  interface etc. has until now been encoded in the device URI, e.g.
  ``xnvme_dev_open("/dev/nvme0n1?async=io_uring")`` in order to use
  ``io_uring``, this has now been replaced by ``struct xnvme_opts``, making it
  much easier to instrument the library runtime via the API. The command-line
  is also affected, as the command-line parser is extended enabling parsing of
  said options along with the tests, examples, and tools are extended with
  these options.

* Device enumeration populated a list with device-identifiers, this has been
  replaced by invocation of a user-defined call-back function for each
  discovered device. Where instead of identifiers, device-handles are passed
  to the callback. This makes it much simpler to e.g. filtering namespace with
  a specific command-set.

* To support the above then most of information carried in the ``xnvme_ident``
  is removed, expect for the ``uri``, and extended with: ``dtype``, ``nsid``,
  and ``csi``. Where ``dtype`` denotes e.g. ``file``, ``block device``, ``NVMe
  controller``, ``NVMe Namespace``.

* The ``xnvme_znd_mgmt_send()`` has now has an explicit ``select_all`` argument
  for setting the matching command-field, this replaces the use of the
  non-standardized ``zrasf`` field associated enumeration-values.

* Documentation for building on Gentoo is added along with addition to the
  automated build-test.

* nvme:spec: expanded with PCIe-bar registers

* Support for enumeration and device-handles for Linux NVMe Namespaces
  represented in devfs as char-devices, e.g. ``/dev/ngXnY`` is added.

* **Experimental** support for sending NVMe commands over ``io_uring``
  infrastructure is added. Think of this as sending the **synchronous** NVMe
  Driver ``ioctl()`` commands via the **asynchronous** ``io_uring`` interface.
  You thus get the control and capabilities of the ioctl() with the efficiency
  of ``io_uring``.
  This feature is enabled by setting ``opts.async=io_uring_cmd`` via the API or
  ``--async=io_uring_cmd`` via the command-line. The feature is experimental as
  it depends on non-upstream Kernel Support.

v0.0.24
-------

A release primarily of fixes, a new thread-pool based async. implementation and
a third-party update of fio.

* Third-party
  - fio, updated to 3.27

* Backends
  - posix:async:thrpool: add async-implementation with async.emulation via
    threadpool processing

* A good handful of fixes, see the commit-messages for details

v0.0.23
-------

This release contains updates to third-party repositories along with any
changes necessary for xNVMe due to third-party changes.

* Third-party
  - SPDK updated to v21.04
  - liburing updated to v2.0
  - fio, not updated, due to a compiler-warning breaking the xNVMe build

This release contains another major refactoring of the API along with a handful
of fixes and updates. The goal of the refactoring is to further simplify the
"core" of the API.

* The buffer-allocator ``xnvme_buf_alloc()`` automatically selects the type of
  memory-allocator to use based on the device. However, it took a 'phys'
  argument which is only valid for very specific use-cases. Thus, this argument
  is removed and replaced by explicit ``physical`` allocators. This simplifies
  the "core" usage, without sacrificing low-level control, it is just provided
  via an explitcit interface instead.

* xNVMe now provides an API for file-system file-io
  - Plugs into the synchronous as well as the asynchronous xNVMe command API
  - I/O provided by ``xnvme_file_pread`` and ``xnvme_file_write``
  - Provides support for diirect and non-direct I/O
  - Two tools are provided utilizing the API ``xdd`` a simplified version of
    ``dd`` and ``xnvme_file`` utilizing sync. and sync. code-paths for
    load/dump/copy of files

* Examples
  - Add minimal examples for command submission and completion

* Backends
  - linux:fs: preliminary support for file-system I/O
  - linux:io_uring now does batched completion-handling
  - linux:io_uring now supports kernel-completion-polling (IOPOLL)
  - linux:io_uring fixes for use auto-handling of register-files
  - spdk now provides core-mask control via ident-uri-options
  - spdk now provides shared-memory group control via ident-uri-options

* A good handful of fixes, see the commit-messages for details

v0.0.22
-------

This release contains a major refactoring of the API along with a handful of
minor fixes. The refactoring goals are to align to existing nomenclature and
simplify usage.

* Reduce to five abstractions: devices, queues, commands, and command-contexts
  - Devices are base handles to NVMe Namespaces and a list of devices are
    retrieved via ``xnvme_enumerate()``, and handles to individual devices
    retrieved via ``xnvme_dev_open()`` and released via ``xnvme_dev_close()``.
  - The abstraction formerly known as an ``asynchronous context`` is now dubbed
    a ``queue``. The ``queue`` now has a ``capacity`` instead of a ``depth``.
  - ``queues`` are created on top of ``devices`` and belong to the device.
  - The definition, submission, and completion of a command is encapsulated in
    a context; the command-context. The command-context replaces the previous
    abstraction named the ``request``.
  - A command can reach a device via a ``queue``, in a deferred / asynchronous
    callback-based manner, or it go via the device in a synchronous / blocking
    manner. Regardless, the command needs a context, and the context is
    retrieved via ``xnvme_cmd_ctx_from_queue()`` or
    ``xnvme_cmd_ctx_from_dev()``.
  - Commands are passed down via ``xnvme_cmd_pass`` for NVMe IO Commands, and
    through ``xnvme_cmd_pass_admin`` for NVMe Admin Commands via the given
    command-context.

* Core API reduction
  - The core xNVMe API as provided by ``libxnvme.h`` it is reduced to a minimal
    interface. Auxilary helpers, convenience functions, and pretty-printers are
    no longer part of the core API but provided via individual header-files
  - The core of the xNVMe API thus consists of
    Device Handling: enumerate, dev_open, dev_close
    Memory: alloc, realloc, free, vtophys, virt_alloc, virt_free
    Queueing: init, term, poke, wait, get_command_ctx, get_capacity, get_outstanding
    Commands: pass, pass_admin
    Supporting the four abstractions described above
  - The manual allocation of a request-pool / command-context-pool is no longer
    needed. xNVMe does not prevent you from creating one if you want to, but it
    is no longer required. Each 'queue' now provides a pre-allocated pool of
    resources, and the manual request-pool is thus replaced by a call to the
    function ``xnvme_cmd_ctx_from_queue()``. If you are familiar with
    ``io_uring`` then think of this function as the equivalent of
    ``io_uring_get_sqe()``.

* API re-organization
  - Previously each command-set had its own top-level namespace, e.g. functions
    and structures for the Zoned Command-Set was using ``znd_*``. This was
    slightly quirky since it still relied on core of the xNVMe namespace
    ``xnvme_*`` for device handles etc. Thus, the command-set specific APIs
    providing helper-functions and convenience are now nested in the xNVMe API
    Namespace e.g. ``znd_*`` is now ``xnvme_znd_*`` and provided via
    ``libxnvme_znd.h``.
  - The NVM Command-Set API was ``lblk_*`` it is now ``xnvme_nvm_*``, and
    provided via ``libxnvme_nvm.h``.

* be:linux: changed error-mapping for non-NVMe errors
  - The Linux block based and sync. interfaces does not provide the underlying
    NVMe command status code and status code type since this is hidden behind
    the block-interface. Previously, the NVMe-completion status-code was just
    assigned the ``errno`` provided by the Kernel, which is highly confusing.
    This behavior is replaced by assigning the status-code-type of
    "vendor-specific" to indicate the status-codes are not defined in the spec.

* be:linux:aio: fixed submission and completion paths
  - The submission, via ``cmd_io()``, of a single command would submit all
    outstanding command, effectively limiting queue-depth
  - The completion via ``poke()``/``wait()`` could potentially complete more
    than requested by the user
  - The encapsulation of io-control-blocks, array of io-control-block pointers,
    were all pointing to the same control-block. Note, this was not causing
    issues due to the short-coming in ``cmd_io()``.

v0.0.21
-------

* Refactored backend interface

  - Changed to support interchangeable ``sync`` and ``async`` implementations

* The Linux backend ``be::linux``
  - Merged ``be:lioc``, ``be:laio``, ``be:liou``, and ``nil`` into one backend
    ``be:linux``, having the async-implementation be an engine parameter
    controllable via uri-opt ``?async`` values: ``thr``, ``aio``, ``iou``,
    ``nil``.
  - Added proper support for the Linux Block Device model, replacing the
    ``?pseudo`` option with ``sync`` interfaces ``nvme_ioctl`` and
    ``block_ioctl``. Gracefully falling back to the Block Layer when the given
    device is not an NVMe device, and thus supporting everything the Linux
    Block Supports including the Zoned Block Device model
  - Added support for ``XNVME_CMD_ASYNC`` for ``ioctl``-driven commands. This
    provides an async.interface to Linux driver-ioctls(), for commands other
    than read/write.  Next step is to make it run fast by providing a less
    costly kernel path. This path is enabled via ``?async=thr``.
  - With these changes, the build-configuration of backends has changed and
    documentation describes how to enable/disable the different backends, sync,
    and async implementations

* Changed command behavior

  - api-functions taking command-options, e.g.  ``xnvme_cmd_pass``,
    ``znd_cmd_mgmt_send``, now **require** that either ``XNVME_CMD_SYNC`` or
    ``XNVME_CMD_ASYNC`` is given as argument. When none is given, negated
    ``EINVAL`` is returned.

* xNVMe fio io-engine

  - Replace ``--be`` option with ``--async``, this makes it a easier to
    instrument ``fio`` to use a different async. implementation in the Linux
    backend of xNVMe. Previously it relied on schema-prefix, the prefix-prefix
    was annoying to use with fio as it required escape-chars.

  - ``fio`` scripts and docs have been updated with the new ``--async`` argument

  - ``fio`` scripts simplified and aligned such that they all three can be used
    in the same manner using the ``--sector=default`` and ``--sector=override``
    to override ``rw``, ``iodepth``, and ``bs`` via environment variables.

* Third-party libraries

  - Added Linux/UAPI version to ``xnvme library-info``, this can give a good
    hint on why certain features aren't behaving as expected, such as the Linux
    versions without the Zoned Block headers
  - Updated to fio/v3.23

* A general handful of code-cleanups and fixes, both on style as well as
  potential issues such local-vars shadowing global-vars, potential arithmetic
  overflows

* Continous Integration

  - Added testing of Linux paths using Nullblock instances in addition to
    emulated NVMe devices

  - Added integration of GitHUB/CodeQL, since Semmle got acquired by GitHUB,
    this will replace the lgtm.com integration.

v0.0.20
-------

* Third-party libraries

  - Updated to fio/v3.22
  - Made fio available to the third-party SPDK build
  - Added build of SPDK fio io-engine
  - Fixed missing update of third-party version-strings

* The xNVMe fio io-engine

  - Several fixes to locking/serialization and error-handling
  - Adjusted to changes in upstream ZBD support
  - Changed the zoned fio-example to not be timebased, since it could lead to
    the verify-job never getting to the verify-part when running on emulated
    devices
  - Increased ``ramp_time`` in comparison-script
  - Fixed memory issue due to missing ``get_file_size``

* Backends

  - Added a backend ``nwrp`` the NULL-Async-IO backend, purpose of which is to
    troubleshoot and benchmark the async-io path

* General

  - A bunch of fixes including bad format-strings, out-of-bound / array
    overflows, non-atomic locks, improper error-path handling

* CI

  - Added workflow generating docker-image with latest source, providing
    everything needed to build xNVMe and latest qemu to deploy and experiment
    with xNVMe on emulated NVMe devices
  - Added workflow doing Coverity scan and uploading results for analysis
  - Added ``fio`` binary and SPDK fio io-engines as artifacts. During testing,
    fio is needed, however, the test-environment might not have the same
    version available as the io-engines are built against, usually xNVMe is
    built against the latest release which might not have made it into the
    package repos.

v0.0.19
-------

* Third-party libraries

  - Updated to liburing/v0.7, SPDK/v20.07, fio/v3.21
  - Updated docs describing new third-party requirements for building
  - Adjusted patches and build-system to changes

* Fabrics: SPDK-patches enabling zone-changes over Fabrics

* Added public-domain CI

  - Primarily using GitHUB Actions / Workflows
  - Aux. analysis via lgtm.com
  - Updated docs and scripts for CI via GitHUB Actions

* Updated support for the NVMe Simple-Copy-Command (SCC)

  - Targeting TP 2020.05.04 (Ratified)
  - Added ``tests/scc.c`` testing for SCC-support, print identify fields, and
    exercises the command itself

v0.0.18
-------

* Third-party libraries: SPDK

  - Updated tracking of SPDK to current master(7dbaf54bf) and adjusted linkage
  - Removed patches that are now upstream
  - Updated nvmf/IOCS support

* Fixed non-IOCS device identification

v0.0.17
-------

* Third Party libraries

  - The organization of these has changed such that tracking them and applying
    patches is easier
  - The versions / git-revision info from bundled libraries bundled can now be
    queried via the api calls 'xnvme_3p_ver_*()'
  - The CLI tool 'xnvme' produces these upon request via 'xnvme library-info'
  - Most of the third-party libraries have been updated to, at the time of
    writing, latest versions

* The xNVMe fio IO engine

  - It now supports Zoned Devices!
    It does so by mapping the Zoned Command Set to the ZBD Kernel abstraction
  - It now supports multiple devices!
    Minor caveat; when using multiple-devices then one cannot mix backends
  - The engine was developed against fio-3.20, other versions might pose issues
    with the IO-engine interface leading to segfaults when running or just
    exiting. It should now produce a meaningful error-message when this
    happens.

* be:liou, the io_uring backend

  - Added opcode-checking via the "new" probing feature
  - Replaced READV/WRITEV with READ/WRITE
  - Build of ``be::liou`` on Alpine Linux

* Added ``be::laio`` the Linux/libaio backend

  - A great supplement to the IOCTL, io_uring, and SPDK backends

* Added initial support for NVMe-oF / Fabrics

  - xnvme_dev_open(): 'uri' argument on the form: "fab:<HOST>:<PORT>?nsid=xyz"
  - xnvme_enumerate(): 'sys_uri' argument on the form "fab:<HOST>:<PORT>"
  - Command-line utility: 'xnvme enum' takes '--uri "fab:<HOST>:<PORT>"'
  - See the "docs/tutorial/fabrics.rst" for details

* Added support for I/O Command Set

  - Convenience functions to retrieve command-set specific identity
  - Misc. definitions in the ``libxnvme_spec.h`` headers
  - Utilization of these via the CLI tools ``xnvme`` and ``lblk``

* Added support for Namespace Types (TP 4056 2020-03-05) [verified]

  - Patched SPDK to allow Command Set Selection
  - Added identifier option "?css=0xHEX" for Controller Configuration

* Added support for the Zoned Command Set

  - Support is encapsulated in the library header 'libznd.h'
  - Convenience functions for Zoned Commands
    For example: znd_cmd_mgmt_send(), znd_cmd_mgmt_send(), znd_cmd_append()
    Helpers for retrieving zone-reports with and with descriptor extensions
  - Support in fio via the xNVMe fio I/O Engine
  - CLI tool 'zoned' for convenient command-line management/inspection of zoned
    devices

* Added handling of extended-LBA

  - Expanded ``geometry`` with ``lba_extended`` informing whether
    extended-LBAs are in effect. That is, when ``flbas.bit4`` is set AND the
    current ``lbaf.ms`` is not zero.
  - Expanded ``geometry`` with ``lba_nbytes``, which will always contain the
    size of an LBA in bytes. When ``lba_extended`` is cleared to zero, then
    ``lba_nbytes`` is ``lbaf.ds``, in bytes, when ``lba_extended`` is set to 1,
    then ``lba_nbytes`` is ``lbaf.ds + lbaf.ms``.
  - When ``lba_extended`` is cleared to 0 then the API I/O helpers expect to be
    passed ``dbuf``, and ``mbuf``. When ``lba_extended`` is set to 1, then the
    API I/O helpers expect ``dbuf`` to contain data and meta-data, and expect
    ``mbuf`` to be ``NULL``.

* And a bunch of fixes
  - xnvmec: fixed errno assignment and decode
  - be: added comment on failed attempt at _blockdevice_geometry()
  - Fixed a build-issue on ARM
  - Updated backend documentation and added link to online docs in README

v0.0.16
-------

* Initial public release of xNVMe
