Soft RDMA over Ethernet (RoCE) Driver

This repository contains a full kernel source tree, with the RoCE driver code located in the directory drivers/infiniband/hw/rxe.

Source

Kernel Space Driver

• Github: https://github.com/SoftRoCE/rxe-dev.git (this repository)
• Active Branch: rxe_submission_v18

User Space Library

• Github: https://github.com/SoftRoCE/librxe-dev.git
• Current Version: librxe-1.0.0

Build Instructions

Compile and install kernel:

0. Clone kernel git: 0. git clone https://github.com/SoftRoCE/rxe-dev.git 0. Checkout branch 'rxe_submission_v18'
1. Compile kernel: 0. Enter the source directory cd rxe-dev 0. cp /boot/config-$(uname –r) .config 0. make menuconfig 0. Need to enable CONFIG_RDMA_RXE 0. Need to enable CONFIG_INFINIBAND_ADDR_TRANS=y and CONFIG_INFINIBAND_ADDR_TRANS_CONFIGFS=y in new config file .config 0. make –j 32 0. make modules_install 0. make install 0. make headers_install INSTALL_HDR_PATH=/usr
2. Verify that the new kernel entry is added (e.g. to grub); if not, need to add it manually.
3. Boot with new kernel.

Install user space library (librxe):

0. Install the following package (example shown using RedHat): 0. yum install perl-Switch (name might vary according to distribution)
1. Make sure that the following upstream user space libraries are installed: 0. libibverbs 0. libibverbs-devel 0. libibverbs-utils 0. librdmacm 0. librdmacm-devel 0. librdmacm-utils ( For ubuntu libibverbs-dev libswitch-perl rdmacm-utils ibverbs-utils )
2. Compile and install user space library librxe: 0. git clone https://github.com/SoftRoCE/librxe-dev.git 0. cd librxe-dev 0. ./configure --libdir=/usr/lib64/ --prefix= 0. make 0. make install

Configure Soft-RoCE (RXE):

0. Load ib_rxe kernel module using the rxe_cfg script included in the librxe RPM: 0. rxe_cfg start (this might require sudo or root privileges)
1. Create RXE device over network interface (e.g. eth0): 0. rxe_cfg add eth0
2. Use the status command to display the current configuration: 0. rxe_cfg status
3. If configured successfully, you should see output similar to the following:

    Name  Link  Driver   Speed  NMTU  IPv4_addr  RDEV  RMTU         
    eth0  yes   mlx4_en                          rxe0  1024  (3) 

0. If you are using a Mellanox HCA: Need to make sure that the mlx4_ib kernel module is not loaded (modprobe –rv mlx4_ib) in the soft-RoCE machine.
1. Now you have an Infiniband device called “rxe0” that can be used to run any RoCE app.