FDP - animeshtrivedi/notes GitHub Wiki
- libvnvme docs: https://libnvme.readthedocs.io/en/latest/api.html
- FDP code in
nvme-cli: https://github.com/linux-nvme/nvme-cli/tree/master/plugins/fdp - fio code : https://github.com/axboe/fio/blob/master/dataplacement.c
$ qemu-system-x86_64 --device nvme-subsys,help
nvme-subsys options:
fdp.nrg=<uint32> - (default: 1)
fdp.nruh=<uint16> - (default: 0)
fdp.runs=<size> - (default: 100663296)
fdp=<bool> - (default: false)
nqn=<str>
$ qemu-system-x86_64 --device nvme,help
nvme options:
account-failed=<OnOffAuto> - on/off/auto (default: "auto")
account-invalid=<OnOffAuto> - on/off/auto (default: "auto")
acpi-index=<uint32> - (default: 0)
addr=<int32> - Slot and optional function number, example: 06.0 or 06 (default: -1)
aer_max_queued=<uint32> - (default: 64)
aerl=<uint8> - (default: 3)
backend_defaults=<OnOffAuto> - on/off/auto (default: "auto")
bootindex=<int32>
cmb_size_mb=<uint32> - (default: 0)
discard_granularity=<size> - (default: 4294967295)
drive=<str> - Node name or ID of a block device to use as a backend
failover_pair_id=<str>
ioeventfd=<bool> - (default: false)
logical_block_size=<size> - A power of two between 512 B and 2 MiB (default: 0)
max_ioqpairs=<uint32> - (default: 64)
mdts=<uint8> - (default: 7)
min_io_size=<size> - (default: 0)
msix-exclusive-bar=<bool> - (default: false)
msix_qsize=<uint16> - (default: 65)
multifunction=<bool> - on/off (default: false)
num_queues=<uint32> - (default: 0)
opt_io_size=<size> - (default: 0)
physical_block_size=<size> - A power of two between 512 B and 2 MiB (default: 0)
pmrdev=<link<memory-backend>>
rombar=<uint32> - (default: 1)
romfile=<str>
romsize=<uint32> - (default: 4294967295)
serial=<str>
share-rw=<bool> - (default: false)
smart_critical_warning=<uint8>
sriov_max_vfs=<uint8> - (default: 0)
sriov_max_vi_per_vf=<uint8> - (default: 0)
sriov_max_vq_per_vf=<uint8> - (default: 0)
sriov_vi_flexible=<uint16> - (default: 0)
sriov_vq_flexible=<uint16> - (default: 0)
subsys=<link<nvme-subsys>>
use-intel-id=<bool> - (default: false)
vsl=<uint8> - (default: 7)
write-cache=<OnOffAuto> - on/off/auto (default: "auto")
x-pcie-ari-nextfn-1=<bool> - on/off (default: false)
x-pcie-err-unc-mask=<bool> - on/off (default: true)
x-pcie-extcap-init=<bool> - on/off (default: true)
x-pcie-lnksta-dllla=<bool> - on/off (default: true)
zoned.auto_transition=<bool> - (default: true)
zoned.zasl=<uint8> - (default: 0)
$ qemu-system-x86_64 --device nvme-ns,help
nvme-ns options:
account-failed=<OnOffAuto> - on/off/auto (default: "auto")
account-invalid=<OnOffAuto> - on/off/auto (default: "auto")
backend_defaults=<OnOffAuto> - on/off/auto (default: "auto")
bootindex=<int32>
detached=<bool> - (default: false)
discard_granularity=<size> - (default: 4294967295)
drive=<str> - Node name or ID of a block device to use as a backend
eui64-default=<bool> - (default: false)
eui64=<uint64> - (default: 0)
fdp.ruhs=<str>
logical_block_size=<size> - A power of two between 512 B and 2 MiB (default: 0)
mcl=<uint32> - (default: 128)
min_io_size=<size> - (default: 0)
ms=<uint16> - (default: 0)
mset=<uint8> - (default: 0)
msrc=<uint8> - (default: 127)
mssrl=<uint16> - (default: 128)
nguid=<str> - NGUID or "auto" for random value
nsid=<uint32> - (default: 0)
opt_io_size=<size> - (default: 0)
physical_block_size=<size> - A power of two between 512 B and 2 MiB (default: 0)
pi=<uint8> - (default: 0)
pif=<uint8> - (default: 0)
pil=<uint8> - (default: 0)
share-rw=<bool> - (default: false)
shared=<bool> - (default: true)
uuid=<str> - UUID (aka GUID) or "auto" for random value (default)
write-cache=<OnOffAuto> - on/off/auto (default: "auto")
zoned.cross_read=<bool> - (default: false)
zoned.descr_ext_size=<uint32> - (default: 0)
zoned.max_active=<uint32> - (default: 0)
zoned.max_open=<uint32> - (default: 0)
zoned.numzrwa=<uint32> - (default: 0)
zoned.zone_capacity=<size> - (default: 0)
zoned.zone_size=<size> - (default: 134217728)
zoned.zrwafg=<size> - (default: 18446744073709551615)
zoned.zrwas=<size> - (default: 0)
zoned=<bool> - (default: false)
FDP-experiments github repo has all the code and repo.
set -ex
IMG_HOME=/home/atr/
FDP_IMG_HOME=/home/atr/
sudo /home/atr/local/bin/qemu-system-x86_64 \
-name fdpqemu -m 8G --enable-kvm -cpu host -smp 8 \
-hda $IMG_HOME/ubuntu-2404.qcow2 \
-net user,hostfwd=tcp::7777-:22 -net nic \
-device nvme-subsys,id=nvme-subsys0,fdp=on,fdp.runs=96M,fdp.nrg=1,fdp.nruh=16 \
-device nvme,id=nvme0,serial=deadbeef,subsys=nvme-subsys0 \
-drive id=fdp-1,file=$FDP_IMG_HOME/fdp-1.img,format=raw,if=none \
-device nvme-ns,id=fdp-1,drive=fdp-1,nsid=1 \
-drive id=fdp-2,file=$FDP_IMG_HOME/fdp-2.img,format=raw,if=none \
-device nvme-ns,id=fdp-2,drive=fdp-2,nsid=2,fdp.ruhs=1-15
Explanation from :
Because FDP is a feature that is enabled at the Endurance Group level, it is necessary to configure an “NVMe Subsystem” device that will serve as the Endurance Group “container” and configure some FDP values on it. Adding such a device requires the following command line parameter for QEMU.
-device nvme-subsys,id=nvme-subsys0,fdp=on,fdp.runs=96M,fdp.nrg=1,fdp.nruh=16
This configures the endurance group with a reclaim unit nominal size of 96M (fdp.runs), 1 reclaim group (fdp.nrg), and 16 reclaim unit handles (fdp.nruh). The next command configures the controller and sets it up to be linked to the subsystem.
-device nvme,id=nvme0,serial=deadbeef,subsys=nvme-subsys0
It is necessary to create a namespace. The following command configures both the “drive” (file) that holds the data and the emulated NVMe namespace.
-drive id=fdp-1,file=data.img,format=raw,if=none \
-device nvme-ns,id=fdp-1,drive=fdp-1,nsid=1,fdp.ruhs=1-15
At this point, reclaim unit handles 1 through 15 (both inclusive) will be assigned to the namespace. Reclaim unit handle 0 will remain as a controller-specified reclaim unit handle – if other namespaces are added to the subsystem without fdp.ruhs being specified, they will automatically use reclaim unit handle 0. fdp.ruhs accepts a semi-colon (“;”) separated list of identifiers and may include ranges. For example, to assign reclaim unit handle 1, 2, 3 and 8, set fdp.ruhs to “1-3;8” which will yield the same result as “1,2,3,8”.
The example shown creates two namespaces within the same endurance group. Since namespace 1 does not specify fdp.ruhs,the controller will assign reclaim unit handle 0 and make it the controller-specified reclaim unit handle.
atr@u24clean:~$ sudo nvme fdp configs /dev/nvme0 -e 1
FDP Attributes: 0x80
Vendor Specific Size: 0
Number of Reclaim Groups: 1
Number of Reclaim Unit Handles: 16
Number of Namespaces Supported: 256
Reclaim Unit Nominal Size: 100663296
Estimated Reclaim Unit Time Limit: 0
Reclaim Unit Handle List:
[0]: Initially Isolated
[1]: Initially Isolated
[2]: Initially Isolated
[3]: Initially Isolated
[4]: Initially Isolated
[5]: Initially Isolated
[6]: Initially Isolated
[7]: Initially Isolated
[8]: Initially Isolated
[9]: Initially Isolated
[10]: Initially Isolated
[11]: Initially Isolated
[12]: Initially Isolated
[13]: Initially Isolated
[14]: Initially Isolated
[15]: Initially Isolated
atr@u24clean:~$
from https://github.com/axboe/fio/blob/master/examples/uring-cmd-fdp.fio
This assumes the namespace is already configured with FDP support and has at least 8 available reclaim units. So you need to run it on
n2notn1as the default is in the code.
sudo ./fio examples/uring-cmd-fdp.fio
question Why did the code failed? It is going to have an error if not compatible with the FDP model?