How to configure NVMe oF connections - linux-nvme/nvme-cli GitHub Wiki

How to configure NVMe-oF connections (nvme-fabrics.conf)

nvme-cli 3.0 introduced an INI-format configuration file, nvme-fabrics.conf, that tells nvme connect-all and nvme discover which controllers to connect to and with what parameters. It replaces the old config.json (JSON) and discovery.conf (argv-style) files from nvme-cli 2.x -- see Migrating from config.json / discovery.conf below if you're coming from those.

This page is a practical walkthrough. For the full format specification (precedence rules, validation tiers, the host-identity model) see CONFIG.md If you're adding a new key to the format itself (i.e. contributing to nvme-cli), see How to add a new INI configuration parameter instead.

File location

The default file is /etc/nvme/nvme-fabrics.conf, with an optional drop-in directory /etc/nvme/nvme-fabrics.conf.d/*.conf alongside it -- the same foo.conf + foo.conf.d/ layout systemd uses. Both are optional and read independently; an absent file (or absent directory) is just treated as empty, not an error.

/etc/nvme/
    nvme-fabrics.conf       # main file: per-type defaults (+ connections)
    nvme-fabrics.conf.d/    # optional drop-ins, one persona per file
        prod-fabric.conf
        lab-fabric.conf

--config FILE (on nvme connect, nvme connect-all, nvme discover, and the nvme config subcommands below) points at a different main file; its drop-in directory is always <FILE>.d/, derived automatically.

Sections

A connection is described by an optional [Host] block plus one or more endpoint sections:

  • [Host] -- the host identity (hostnqn, hostid, hostsymname, and any credentials shared by every connection in the file). Omit it entirely to connect as the system default identity (/etc/nvme/hostnqn / /etc/nvme/hostid). At most one per file.
  • [Discovery Controller] -- a Discovery Controller to connect to. Repeatable -- one section per DC.
  • [Subsystem] -- an I/O subsystem to connect to, named by nqn =. Repeatable -- one section per subsystem.
  • [Discovery Controller Defaults] / [I/O Controller Defaults] -- default connection parameters (keep-alive-tmo, ctrl-loss-tmo, reconnect-delay, digests, queue counts, ...) applied to every connection of that class unless something more specific overrides them. There is no third, type-agnostic defaults section -- a value that applies to both is written in both.

Within an endpoint section, each controller = line is one path to that endpoint (a ;-separated list of nvme connect-style options). Repeating it expresses multipath. The subsystem's namespaces become reachable through every path listed. Most keys can be overridden per path on a controller = line; the exception is authentication/TLS parameters (see Authentication and TLS below), which stay on the section since they're bound to (hostnqn, subsysnqn), not to a path.

A minimal single-personality example:

# /etc/nvme/nvme-fabrics.conf

[Discovery Controller Defaults]
keep-alive-tmo = 30

[I/O Controller Defaults]
keep-alive-tmo = 5

[Host]
hostsymname = lab-host-01
# hostnqn/hostid omitted -> /etc/nvme/hostnqn, /etc/nvme/hostid

[Discovery Controller]
controller = transport=tcp;traddr=192.168.1.10;trsvcid=8009

[Subsystem]
nqn        = nqn.2024-01.com.example:data.vol1
controller = transport=tcp;traddr=192.168.1.20;trsvcid=4420

[Subsystem]
nqn           = nqn.2024-01.com.example:data.vol2
ctrl-loss-tmo = 1800                                     # per-subsystem override
controller    = transport=tcp;traddr=192.168.1.21;trsvcid=4420
controller    = transport=tcp;traddr=192.168.1.22;trsvcid=4420   # dual-path

Multiple host personalities

A host normally has one identity. If it needs to present different identities to different fabrics give each identity its own drop-in file under nvme-fabrics.conf.d/, opening with its own [Host] block that states hostnqn explicitly:

# /etc/nvme/nvme-fabrics.conf.d/prod-fabric.conf

[Host]
hostnqn     = nqn.2014-08.org.nvmexpress:uuid:1111...-A
hostid      = 46ba5037-7ce5-41fa-9452-48477bf00080
hostsymname = Hydra

[Discovery Controller]
nqn        = nqn.2014-08.org.example:cdc.prod
controller = transport=tcp;traddr=10.0.0.5;trsvcid=8009

A hostnqn may only appear in one file. Reusing it across the main file and a drop-in (or across two drop-ins) will result in validation error.

Precedence

A parameter resolves most-specific-first:

controller = line > endpoint section ([Discovery Controller] / [Subsystem]) > [Host] > type defaults (*Defaults) > kernel default

[Host] deliberately sits above the type defaults. A value in a persona's [Host] wins over even that file's own *Defaults section. Drop-ins are additive: a drop-in's own *Defaults section is a scoped overlay for that file's connections only, and never leaks to the top-level file or to sibling drop-ins.

To hand a parameter back to the kernel's built-in default use an empty assignment, the same idiom a systemd drop-in uses to reset a setting:

[I/O Controller Defaults]
ctrl-loss-tmo = 450

[Subsystem]
nqn           = nqn.2024-01.com.example:data.vol3
ctrl-loss-tmo =                          # reset: let the kernel decide
controller    = transport=tcp;traddr=192.168.1.23;trsvcid=4420

Authentication and TLS credentials

KX-HMAC-CHAP (in-band authentication) and TLS (encryption) settings live on [Host] or an endpoint section -- never on a controller = line, since they're bound to the (hostnqn, subsysnqn) pair, not to a single path:

[Subsystem]
nqn                = nqn.2024-01.com.example:secure.vol1
tls                = true
tls-key            = NVMeTLSkey-1:01:...
kxchap-secret      = DHHC-1:00:...
kxchap-ctrl-secret = DHHC-1:00:...              # bidirectional auth
controller         = transport=tcp;traddr=10.0.0.9;trsvcid=4420

See How to set up inband authentication and How to set up TLS for NVMe-TCP for how to generate these values; both pages' "saving the configuration" sections use nvme config create exactly as shown below.

Building a configuration with nvme config create

Rather than hand-editing the file, nvme config create builds one connection entry from command-line options and merges it into the target file:

nvme config create --transport tcp --traddr 192.168.1.20 --trsvcid 4420 \
    --nqn nqn.2024-01.com.example:data.vol1

Running the same command again is a no-op. Use --discovery to add a Discovery Controller entry instead (in which case --nqn is optional -- it defaults to the well-known discovery NQN), and --host-symname / an explicit --hostnqn to write into a new drop-in for a distinct persona rather than the main file. nvme connect-all (or plain nvme connect, which reads the same file by default) then picks up whatever is saved.

Inspecting and validating

nvme config show        # render the resolved config as equivalent command lines
nvme config show -o json   # ... or as structured JSON
nvme config validate    # parse + validate without connecting to anything

Addressing is shown exactly as configured, with no hostname resolution attempted, and a persona with no explicit hostnqn/hostid is shown without those fields rather than the system default it would resolve to at connect time.

Migrating from config.json / discovery.conf

If you're upgrading from nvme-cli 2.x, the first nvme connect, connect-all, or discover call implicitly converts an existing config.json/discovery.conf for you. To do it explicitly instead:

nvme config convert          # writes /etc/nvme/nvme-fabrics.conf
nvme config status           # check what's left to convert, if anything

On success, each converted legacy file is renamed to <name>.converted (a symlink back to itself) so a repeat run is safe. nvme config status reports the on-disk state of both legacy files without changing anything, which is the quickest way to confirm there's nothing left over after an upgrade.

⚠️ **GitHub.com Fallback** ⚠️