How to configure NVMe oF connections - linux-nvme/nvme-cli GitHub Wiki
nvme-cli 3.0 introduced an INI-format configuration file, nvme-fabrics.conf,
that tells nvme connect-all and nvme discover which controllers to connect
to and with what parameters. It replaces the old config.json (JSON) and
discovery.conf (argv-style) files from nvme-cli 2.x -- see
Migrating from config.json / discovery.conf
below if you're coming from those.
This page is a practical walkthrough. For the full format specification
(precedence rules, validation tiers, the host-identity model) see
CONFIG.md
If you're adding a new key to the format itself (i.e. contributing to
nvme-cli), see
How to add a new INI configuration parameter
instead.
The default file is /etc/nvme/nvme-fabrics.conf, with an optional drop-in
directory /etc/nvme/nvme-fabrics.conf.d/*.conf alongside it -- the same
foo.conf + foo.conf.d/ layout systemd uses. Both are optional and read
independently; an absent file (or absent directory) is just treated as empty,
not an error.
/etc/nvme/
nvme-fabrics.conf # main file: per-type defaults (+ connections)
nvme-fabrics.conf.d/ # optional drop-ins, one persona per file
prod-fabric.conf
lab-fabric.conf
--config FILE (on nvme connect, nvme connect-all, nvme discover, and the
nvme config subcommands below) points at a different main file; its drop-in
directory is always <FILE>.d/, derived automatically.
A connection is described by an optional [Host] block plus one or more
endpoint sections:
-
[Host]-- the host identity (hostnqn,hostid,hostsymname, and any credentials shared by every connection in the file). Omit it entirely to connect as the system default identity (/etc/nvme/hostnqn//etc/nvme/hostid). At most one per file. -
[Discovery Controller]-- a Discovery Controller to connect to. Repeatable -- one section per DC. -
[Subsystem]-- an I/O subsystem to connect to, named bynqn =. Repeatable -- one section per subsystem. -
[Discovery Controller Defaults]/[I/O Controller Defaults]-- default connection parameters (keep-alive-tmo,ctrl-loss-tmo,reconnect-delay, digests, queue counts, ...) applied to every connection of that class unless something more specific overrides them. There is no third, type-agnostic defaults section -- a value that applies to both is written in both.
Within an endpoint section, each controller = line is one path to that
endpoint (a ;-separated list of nvme connect-style options). Repeating it
expresses multipath. The subsystem's namespaces become reachable through every
path listed. Most keys can be overridden per path on a controller = line; the
exception is authentication/TLS parameters (see
Authentication and TLS below), which stay
on the section since they're bound to (hostnqn, subsysnqn), not to a path.
A minimal single-personality example:
# /etc/nvme/nvme-fabrics.conf
[Discovery Controller Defaults]
keep-alive-tmo = 30
[I/O Controller Defaults]
keep-alive-tmo = 5
[Host]
hostsymname = lab-host-01
# hostnqn/hostid omitted -> /etc/nvme/hostnqn, /etc/nvme/hostid
[Discovery Controller]
controller = transport=tcp;traddr=192.168.1.10;trsvcid=8009
[Subsystem]
nqn = nqn.2024-01.com.example:data.vol1
controller = transport=tcp;traddr=192.168.1.20;trsvcid=4420
[Subsystem]
nqn = nqn.2024-01.com.example:data.vol2
ctrl-loss-tmo = 1800 # per-subsystem override
controller = transport=tcp;traddr=192.168.1.21;trsvcid=4420
controller = transport=tcp;traddr=192.168.1.22;trsvcid=4420 # dual-pathA host normally has one identity. If it needs to present different identities
to different fabrics give each identity its own drop-in file under
nvme-fabrics.conf.d/, opening with its own [Host] block that states
hostnqn explicitly:
# /etc/nvme/nvme-fabrics.conf.d/prod-fabric.conf
[Host]
hostnqn = nqn.2014-08.org.nvmexpress:uuid:1111...-A
hostid = 46ba5037-7ce5-41fa-9452-48477bf00080
hostsymname = Hydra
[Discovery Controller]
nqn = nqn.2014-08.org.example:cdc.prod
controller = transport=tcp;traddr=10.0.0.5;trsvcid=8009A hostnqn may only appear in one file. Reusing it across the main file and a
drop-in (or across two drop-ins) will result in validation error.
A parameter resolves most-specific-first:
controller = line > endpoint section ([Discovery Controller] /
[Subsystem]) > [Host] > type defaults (*Defaults) > kernel default
[Host] deliberately sits above the type defaults. A value in a persona's
[Host] wins over even that file's own *Defaults section. Drop-ins are
additive: a drop-in's own *Defaults section is a scoped overlay for that
file's connections only, and never leaks to the top-level file or to sibling
drop-ins.
To hand a parameter back to the kernel's built-in default use an empty assignment, the same idiom a systemd drop-in uses to reset a setting:
[I/O Controller Defaults]
ctrl-loss-tmo = 450
[Subsystem]
nqn = nqn.2024-01.com.example:data.vol3
ctrl-loss-tmo = # reset: let the kernel decide
controller = transport=tcp;traddr=192.168.1.23;trsvcid=4420KX-HMAC-CHAP (in-band authentication) and TLS (encryption) settings live on
[Host] or an endpoint section -- never on a controller = line, since they're
bound to the (hostnqn, subsysnqn) pair, not to a single path:
[Subsystem]
nqn = nqn.2024-01.com.example:secure.vol1
tls = true
tls-key = NVMeTLSkey-1:01:...
kxchap-secret = DHHC-1:00:...
kxchap-ctrl-secret = DHHC-1:00:... # bidirectional auth
controller = transport=tcp;traddr=10.0.0.9;trsvcid=4420See How to set up inband authentication
and How to set up TLS for NVMe-TCP for how to
generate these values; both pages' "saving the configuration" sections use nvme config create exactly as shown below.
Rather than hand-editing the file, nvme config create builds one connection
entry from command-line options and merges it into the target file:
nvme config create --transport tcp --traddr 192.168.1.20 --trsvcid 4420 \
--nqn nqn.2024-01.com.example:data.vol1Running the same command again is a no-op. Use --discovery to add a Discovery
Controller entry instead (in which case --nqn is optional -- it defaults to
the well-known discovery NQN), and --host-symname / an explicit --hostnqn to
write into a new drop-in for a distinct persona rather than the main file. nvme connect-all (or plain nvme connect, which reads the same file by default)
then picks up whatever is saved.
nvme config show # render the resolved config as equivalent command lines
nvme config show -o json # ... or as structured JSON
nvme config validate # parse + validate without connecting to anythingAddressing is shown exactly as configured, with no hostname resolution
attempted, and a persona with no explicit hostnqn/hostid is shown without
those fields rather than the system default it would resolve to at connect time.
If you're upgrading from nvme-cli 2.x, the first nvme connect, connect-all,
or discover call implicitly converts an existing
config.json/discovery.conf for you. To do it explicitly instead:
nvme config convert # writes /etc/nvme/nvme-fabrics.conf
nvme config status # check what's left to convert, if anythingOn success, each converted legacy file is renamed to <name>.converted (a
symlink back to itself) so a repeat run is safe. nvme config status reports
the on-disk state of both legacy files without changing anything, which is the
quickest way to confirm there's nothing left over after an upgrade.