Validate that RXE is working - SoftRoCE/rxe-dev GitHub Wiki
To validate that RXE is working need to do the following steps:
Validate that kernel modules are loaded using lsmod:
1- Check that RXE drive is loaded:
# lsmod | grep rxe
rdma_rxe 114688 2
ib_core 114688 13 rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_ib,mlx5_ib,ib_mad,rdma_rxe,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib
ip6_udp_tunnel 16384 2 vxlan,rdma_rxe
udp_tunnel 16384 2 vxlan,rdma_rxe
2- Check that ib_uverbs is loaded:
# lsmod | grep ib_uverbs
ib_uverbs 47211 0
- If the RXE driver is not loaded use rxe_cfg script that comes with librxe (userspace library) to load it.
Validate that userspace applications are working:
1- Check that RXE devices are available using ibv_devices (included in libibvers-utils):
# ibv_devices
device node GUID
------ ----------------
rxe0 0202c9fffe07d511
2- Check that RXE device can be accessed using ibv_devinfo (included in libibverbs-utils) and need to see that port state is in PORT_ACTIVE:
# ibv_devinfo -d rxe0
hca_id: rxe0
transport: InfiniBand (0)
fw_ver: 0.0.0
node_guid: 0202:c9ff:fe07:d511
sys_image_guid: 0000:0000:0000:0000
vendor_id: 0x0000
vendor_part_id: 0
hw_ver: 0x0
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
Validate that traffic over RXE device is working:
Send traffic using ibv_*_pingpong tests (included in libibverbs-utils):
server:
# ibv_rc_pingpong -g 0 -d rxe0 -i 1
local address: LID 0x0000, QPN 0x000011, PSN 0x660fec, GID fe80::202:c9ff:fe07:d511
remote address: LID 0x0000, QPN 0x000011, PSN 0x1c8285, GID fe80::202:c9ff:fe07:d411
8192000 bytes in 0.03 seconds = 1894.16 Mbit/sec
1000 iters in 0.03 seconds = 34.60 usec/iter
Client:
# ibv_rc_pingpong -g 0 -d rxe0 -i 1 10.7.17.88
local address: LID 0x0000, QPN 0x000011, PSN 0x1c8285, GID fe80::202:c9ff:fe07:d411
remote address: LID 0x0000, QPN 0x000011, PSN 0x660fec, GID fe80::202:c9ff:fe07:d511
8192000 bytes in 0.03 seconds = 1888.59 Mbit/sec
1000 iters in 0.03 seconds = 34.70 usec/iter
Send traffic using rping (RDMACM):
server:
# rping -s -a 11.135.196.6 -v -C 10
server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
server ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
server ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
server ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
server ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
server ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
server ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
server ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA
server DISCONNECT EVENT...
wait for RDMA_READ_ADV state 10
client:
# rping -c -a 11.135.196.6 -v -C 10
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA
client DISCONNECT EVENT...