Connect to the NVMf frontend volume - longhorn/longhorn-tests GitHub Wiki

0. Summary

This document describes how to connect to a Longhorn nvmf frontend v2 volume.

  • From NVMf volume engine, we can get endpoint information as below
    • nvmf://{IP}:{Port}/{NQN}
  • Connect to the NVMf device on host machine
    • nvme connect -t tcp -a {IP} -s {Port} -n {NQN}
  • Disconnect to the NVMf device on host machine
    • nvme disconnect -n {NQN}

1. Setup

  • OS: SLES15 SP6
  • Required package: nvme-cli zypper install nvme-cli
  • Longhorn deployed
  • Enable v2 data engine
  • Create a v2 volume with nvmf frontend
  • Attach to node, wait volume attached

2. Get volume endpoint

  • Use command kl get engine -o yaml | grep endpoint:
> kl get engine
NAME       DATA ENGINE   STATE     NODE               INSTANCEMANAGER                                     IMAGE                                              AGE
vol1-e-0   v2            running   ip-172-31-38-206   instance-manager-8098febec436ca87e7072170b29e9c99   longhornio/longhorn-instance-manager:master-head   23m

> kl get engine vol1-e-0 -o yaml | grep endpoint:
  endpoint: nvmf://10.42.0.30:20006/nqn.2023-01.io.longhorn.spdk:vol1-e-0
  • nvmf://10.42.0.30:20006/nqn.2023-01.io.longhorn.spdk:vol1-e-0 is the endpoint

3. Connect to NVMe device

  • Parse the endpoint
    • nvmf://10.42.0.30:20006/nqn.2023-01.io.longhorn.spdk:vol1-e-0
    • target IP: 10.42.0.30
    • port: 20006
    • NQN(NVMe Qualified Name): nqn.2023-01.io.longhorn.spdk:vol1-e-0
  • Connect command: nvme connect -t tcp -a {target IP} -s {port} -n {NQN}
    • example
    > nvme connect -t tcp -a 10.42.0.30 -s 20006 -n nqn.2023-01.io.longhorn.spdk:vol1-e-0
>
> connecting to device: nvme0
  • List device: nvme list
    • Get /dev/nvme0n1 as result
    > nvme list 
Node                  Generic               SN                   Model                                    Namespace  Usage                      
Format           FW Rev  
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            00000000000000000000 SPDK bdev Controller                     0x1         21.47  GB /  21.47  GB    512   B +  0 B   24.09.1
    Also can find in command lsblk 
    > lsblk | grep nvme0n1
nvme0n1 259:1    0   20G  0 disk

4. Disconnect the NVMe device

  • disconnect command: nvme disconnect -n {NQN}
    • After logout, the /dev/nvme0n1 disappear
    > nvme disconnect -n nqn.2023-01.io.longhorn.spdk:vol1-e-0
NQN:nqn.2023-01.io.longhorn.spdk:vol1-e-0 disconnected 1 controller(s)
>
> nvme list
Node                  Generic               SN                   Model                                    Namespace  Usage                      
Format           FW Rev  
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------