LFX weekly update - Aditya-Sarna/lfx_weeklyupdate GitHub Wiki

Weekly update

I have been working on Anssi’s tasks for the unlabeled rack server SERVER DPU HOST ? (DPU host 5) and its NVIDIA BlueField-3. I checked the lab repository first, then verified the live environment.

Task Status
1. Identify and verify the devices Done (live)
2. SSH to the host and the DPU Done (host via iDRAC / 172.22.222.14; DPU via rshim)
3. Persistent management IPs Prepared. MACs collected. No production edit. PR not opened yet
4. DOCA / PCIe / doca_dma_copy Partial. Host DOCA 3.4.0 and DPU 2.6.0 measured. mst and doca_dma_copy not run

1. Identify and verify the devices

Documentation

Documentation/ips.md and ansible/inventory assign these planned addresses. iDRAC is the BMC; DPU Host 5 is the Ubuntu host. Those are different machines from a login point of view, on the same chassis.

Role Name in docs Planned address
DH5 iDRAC (BMC) DH5 iDRAC 172.22.2.5
DPU Host 5 (Ubuntu) DPU Host 5 / dh5 172.22.1.5
BlueField-3 Nvidia BF3 / bf3 172.22.3.5
BlueField-3 BMC Nvidia BF3 bmc / bf3bmc 172.22.4.5

Inventory lists dh5 at 172.22.1.5 and bf3 at 172.22.3.5. There is no dh5bmc entry for 172.22.2.5.

The cabling map labels one chassis SERVER DPU HOST ?, dh5, Dell R760xa, and Nvidia BlueField3, with iDRAC 172.22.2.5. hardware/dh5/README.md is a stub (Dell R760ax, “Blue Filed 3”) and has no PCI or rshim notes. The working rshim example in the repository is for BlueField-2 on dh2.

Live identity

The unlabeled rack name and DPU Host 5 are the same machine.

  • iDRAC at 172.22.2.5 reports PowerEdge R760xa, service tag 7GDGN34, DNS name idrac-7GDGN34.
  • The host console shows dh5 login:. hostname returns dh5.
  • The host OS is Ubuntu 24.04.3 LTS.

How to connect

I did not use a dh5-specific 1Password item. Credentials that already work on the other Dell lab servers worked for iDRAC root and host root.

  1. Connect to the lab VPN.
  2. Open iDRAC: ssh [email protected] . racadm getsysinfo shows model, service tag, and power state.
  3. Use Virtual Console on the iDRAC dashboard and log in as root on dh5.
  4. The host management address in use is 172.22.222.14 on eno12399np0. From the VPN, TCP port 22 on that address accepts connections. The planned address 172.22.1.5 does not answer.

BlueField-3 on the host

lspci on dh5:

PCI Function
0d:00.0 BlueField-3 integrated ConnectX-7
0d:00.1 BlueField-3 integrated ConnectX-7
0d:00.2 BlueField-3 SoC Management Interface

Host-side data interfaces: enp13s0f0np0 172.16.3.65, enp13s0f1np1 172.16.3.66.

After the rshim service is running, /dev/rshim0/misc reports BlueField-3 (Rev 1) at pcie-0000:0d:00.2.

Documentation compared with the lab

Documentation Lab
Planned host 172.22.1.5 DHCP assigned 172.22.222.14 (unknown pool)
Planned BF3 172.22.3.5 Not reachable from the VPN
Host OS Ubuntu 22.04 Host is Ubuntu 24.04.3
R760 / R760ax / R760xa R760xa
Map “Server DPU Host ?” Same chassis as dh5
dh5bmc in inventory Missing
DHCP reservations for dh5 / BF3 Missing in the repository dhcpd.conf
Map host IP 172.22.1.5 Not drawn. The DH5 chassis on the map is labeled 172.22.1.4, which ips.md assigns to DPU Host 4

iDRAC MAC: a8:3c:a5:38:19:f8.

Host NICs from iDRAC: D4:04:E6:D8:0C:70, D4:04:E6:D8:0C:71, C4:CB:E1:CD:56:C8, C4:CB:E1:CD:56:C9.

2. Verify SSH access to the host and the DPU

Host

SSH to iDRAC [email protected] works (racadm>>). That is the BMC, not the host OS.

Login as root on dh5 works from the iDRAC Virtual Console. From the VPN, 172.22.222.14 answers on TCP port 22 (SSH offered publickey,password). I also used password SSH from my laptop to that address. Planned host 172.22.1.5 remains unused.

DPU

SSH to 172.22.3.5 from the VPN fails. The working path is rshim from the host, as documented for dh2.

The rshim package was already installed (2.7.3-0.g17aaf11.2604086). Host apt sources include the NVIDIA DOCA 3.4.0 repository. The rshim service was not running, so /dev/rshim0 was missing. modprobe rshim fails; this rshim is a userspace service kernel module.

systemctl enable --now rshim
ip link set tmfifo_net0 up
ip addr add 192.168.100.1/30 dev tmfifo_net0
ping -c 2 192.168.100.2
ssh [email protected]

Ping to 192.168.100.2 succeeds. ssh [email protected] succeeds. The DPU prompt is ubuntu@localhost. Inventory lists bf3 as ansible_user=root at 172.22.3.5; the working login is ubuntu over rshim, not root at 172.22.3.5.

Addresses used

Connection Address Interface
iDRAC 172.22.2.5 iDRAC BMC
Host management 172.22.222.14 eno12399np0
Host to DPU (rshim) 192.168.100.1 / 192.168.100.2 tmfifo_net0
Planned DPU LAN 172.22.3.5 not in use

On the DPU, hostname is localhost.localdomain. The login banner is Ubuntu 22.04.3 LTS (5.15.0-1035-bluefield, aarch64). /etc/mlnx-release is DOCA_2.6.0_BSP_4.6.0_Ubuntu_22.04-5.24-01.prod. tmfifo_net0 is 192.168.100.2/30. At ip -br a, enp3s0f0s0 was 172.16.3.92/24 and enp3s0f1s0 was 172.16.3.107/24. oob_net0 was UP with no IPv4 on that listing. An earlier login banner showed oob_net0 172.22.222.18; that address did not answer ICMP or SSH from the VPN.

3. Persistent management IPs

I have not changed live dhcpd on 172.22.0.1. I only prepared reservations from measured MACs, for a later branch/PR.

How IPs are assigned

The lab uses ISC dhcpd on 172.22.0.0/16. The copy in this repository is hardware/mgmt/fs/etc/dhcp/dhcpd.conf.

  1. Reservations: host blocks with hardware ethernet and fixed-address (dh1–dh4, BF2, iDRACs, ToR).
  2. Unknown pool: range 172.22.222.1 172.22.222.254. Docs call 172.22.222.0/24 “dhcp allocated to any unknown device.”

Shared options: domain lab.opiproject.org, DNS 8.8.8.8 / 1.1.1.1, router 172.22.0.1.

The repository file is not proven identical to production. DH5 iDRAC already holds the planned 172.22.2.5 with DHCP enabled, but there is no matching host block in the repo file.

What each address is doing now

Role Planned Live How it got the address
DH5 iDRAC 172.22.2.5 172.22.2.5 DHCP on iDRAC. MAC a8:3c:a5:38:19:f8. No matching host in repo dhcpd.conf
DPU Host 5 172.22.1.5 172.22.222.14 on eno12399np0 Unknown pool. No host dh5 in repo. 172.22.1.5 does not answer
BlueField-3 172.22.3.5 rshim 192.168.100.2; OOB once showed 172.22.222.18 No reservation. 172.22.3.5 does not answer
BlueField-3 BMC 172.22.4.5 Not seen No reservation

rshim 192.168.100.1/30 is a host–DPU link, not DHCP.

MACs I measured (2026-09-07 on root@dh5)

eno12399np0      UP             d4:04:e6:d8:0c:70
oob_net0         UP             9c:63:c0:e4:b7:f4

eno12399np0 matches iDRAC NIC.Integrated.1-1-1 (D4:04:E6:D8:0C:70).

I am not using the ToR Ethernet30 LLDP MAC d4:04:e6:2a:b0:21 (adjacent to repo host dh4 d4:04:e6:2a:b0:20).

Port MAC Proposed host / address
eno12399np0 (DPU Host 5) d4:04:e6:d8:0c:70 dh5 → 172.22.1.5
DH5 iDRAC a8:3c:a5:38:19:f8 dh5idrac → 172.22.2.5
BF3 oob_net0 9c:63:c0:e4:b7:f4 dh5bf3 → 172.22.3.5
BF3 BMC unknown omit dh5bf3bmc until we have a MAC

Prepared reservation text (not applied)

host dh5 {
    hardware ethernet d4:04:e6:d8:0c:70;
    fixed-address 172.22.1.5;
}

host dh5idrac {
    hardware ethernet a8:3c:a5:38:19:f8;
    fixed-address 172.22.2.5;
}

host dh5bf3 {
    hardware ethernet 9c:63:c0:e4:b7:f4;
    fixed-address 172.22.3.5;
}

I also need to add dh5bmc to ansible/inventory [hostbmcs] as 172.22.2.5 (inventory, not dhcpd). Next step is a branch and PR. After merge, renew DHCP on the host and DPU and confirm 172.22.1.5 / 172.22.3.5.

4. BlueField-3 host and DPU setup

I checked installed versions on both sides. I have not run mst, flint, or doca_dma_copy. I have not flashed anything.

Host DOCA / NVIDIA drivers (root@dh5, 2026-09-07)

Linux dh5 6.8.0-138-generic #138-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 31 22:41:49 UTC 2026 x86_64

Ubuntu 24.04.3. Apt source:

deb [signed-by=/etc/apt/trusted.gpg.d/nvidia-doca-debian-gpg-public-key.gpg] https://linux.mellanox.com/public/repo/doca/3.4.0/ubuntu24.04/x86_64/ ./

DOCA-Host 3.4.0 is installed, not only configured as a repo. Metapackages: doca-all / doca-ofed / doca-devel / doca-runtime 3.4.0-085000. Tools and SDK packages are 3.4.0112-1, including doca-sdk-dma and doca-samples. doca-perftest is 3.0.15-1. The dpkg listing stopped at doca-sdk-mgmt; ofed_info, host NIC firmware, and remaining mlnx / mft lines were not captured.

rshim was already present: 2.7.3-0.g17aaf11.2604086.

DPU DOCA / BF-Bundle

Item Live
OS Ubuntu 22.04.3, 5.15.0-1035-bluefield, aarch64
/etc/mlnx-release DOCA_2.6.0_BSP_4.6.0_Ubuntu_22.04-5.24-01.prod
bfver / flint / DPU dpkg Not captured

Which versions belong together

NVIDIA’s rule is to pair DOCA-Host and BF-Bundle from the same DOCA Downloads row.

Side Live Same-release pair?
Host DOCA-Host 3.4.0-085000 (doca-all) No
DPU 2.6.0 / BSP 4.6.0 / Ubuntu 22.04 No
Paper pin from task 5 3.5.0 on both sides Not installed

Host–DPU management

rshim ping and SSH work (192.168.100.1 ↔ 192.168.100.2). Planned 172.22.3.5 does not answer.

Still open on task 4

  • Host: ofed_info, doca_caps, ethtool NIC firmware, mst status
  • DPU: bfver, DPU dpkg, doca_dma sample path
  • doca_dma_copy not run. Host already has doca-sdk-dma and doca-samples at 3.4; DPU is 2.6, so a cross-version run may fail.