Reference DMA - GregWatson/NetFPGA-PLUS-GW GitHub Wiki

reference_dma

hw/projects/reference_dma

• NetFPGA-PLUS Specific cores
  • xilinx open-nic-shell
  • nf_data_sink

• Xilinx open-nic-driver

This Reference DMA module was developed to measure accurately the DMA performance in the outbound (host to card) direction.

The Xilinx-provided QDMA subsystem in open-nic-shell serves as a DMA engine for the NetFPGA PLUS reference designs. It includes QDMA IP, a DMA engine and AXI4 Interconnect module. The open-nic-driver manages the QDMA module which handles the transfer of packets from host memory across the PCIe bus to the card memory. Once a packet is in the card the QDMA tries to send it to the CMAC (Ethernet) ports as fast as possible. This Reference DMA design uses an instance of the nf_data_sink module which measures the observed rate of data transferred from the card DMA to the CMAC ports.

The nf_data_sink module has a passive (observer) interface that connects to an AXI4 interface at the output of the DMA. As such, nf_data_sink could be used to monitor the performance of any AXI4 interface.

1. Make sure you clone the latest version of the NetFPGA package. Please ensure that you have the necessary packages installed. The current testing infrastructure is Python based.

$ git clone https://github.com/NetFPGA/NetFPGA-PLUS.git

2. Make sure the ${NFPLUS_FOLDER}/tools/settings.sh file is updated in your system.

Check if the variables like NFPLUS_FOLDER, NF_DESIGN_DIR, BOARD_NAME are set in the right path, e.g.:

export NF_PROJECT_NAME=reference_dma

To set up the environment variables, source both relevant settings files, e.g.:

source ~/NetFPGA-PLUS/tools/settings.sh
source /tools/Xilinx/Vivado/2023.2/settings64.sh

3. Do a make only the first time when you have a fresh repository. It is not necessary to repeat this make every time you want to build a new project unless you have made any changes to the IP cores. This will build all cores, including nf_data_sink.

[root@nf-test109 ~]# cd $NFPLUS_FOLDER 
[root@nf-test109 NetFPGA-PLUS]# make 

4. Please follow the CAM instructions in the Building Your First Project page. Note: The CAM core must be generated prior to running this project.

5.Compile the project to create the required bitfile reference_dma.bit.

$ make -C $NF_DESIGN_DIR/hw

6. Program the FPGA with the design bitfile using xsdb (Xilinx tool).

To create your own bitfile and run the Hardware tests:

[root@nf-test109 reference_dma]# make
[root@nf-test109 reference_dma]# cd bitfiles
[root@nf-test109 bitfiles]# xsdb

On the xsdb console, use fpga -f reference_switch.bit to program the FPGA with the bitfile. Reboot the machine (do NOT power cycle).

7. Check if the bit file is loaded using the following command.

[root@nf-test109 scripts]# lspci –vxx | grep Xilinx

If you don't see the Xilinx device, you need to reprogram the FPGA and reboot as mentioned earlier.

8. Build the Linux driver for the NetFPGA PLUS board.

[root@nf-test109 ]# cd ${NFPLUS_FOLDER}/sw/driver/
[root@nf-test109 ]# make
[root@nf-test109 ]# insmod open-nic-driver/onic.ko

To compile the driver you must have the correct header files installed that match your kernel.

• Run lsmod | grep onic which should print the word "onic" if the onic driver has been loaded.

Then run ifconfig -a to check if you are able to see the 'nfX' interfaces.

9. Running the test

• Compile the test application. This is in NetFPGA-PLUS/sw/app/pkt_send.

[root@nf-test109 ]# cd $NFPLUS_FOLDER/sw/app/pkt_send
[root@nf-test109 ]# make

will generate a binary called pkt_send.

• Run the test app. For example: use the app to see the actual clock frequencies used on the card:

[root@nf-test109 ]# sudo ./pkt_send -z
AXI clock:  339.27 MHz
AXIS clock: 339.27 MHz

Or get the performance for a particular packet size:

[root@nf-test109 ]# sudo ./pkt_send -b 1000 -n 100000
Performance was 25182.5 Mbps ( 25.18 Gbps)

• Use the -h option to see how to use the app.

You can find more information related to hardware and simulation tests here:

• Simulation tests
• HW tests.

The test infrastructure is based on Python. You can find the tests inside the projects/{project_name}/test folder.