ROACH_10GbE_tutorial - david-macmahon/wiki_convert_test GitHub Wiki

This page will teach you how to use the 10GbE_v2 Simulink interface on ROACH. It will send packets out through one interface and be received by another.

PLEASE NOTE THAT THIS TUTORIAL HAS BEEN LARGELY SUPERCEDED BY THE UPDATED SECOND TUTORIAL FROM THE 2009 WORKSHOP. WE RECOMMEND YOU USE THAT ONE INSTEAD. THIS PAGE IS LEFT HERE FOR COMPLETENESS AND REFERENCE ONLY. Note that there is also a bug in the versions of tcpborphserver, tgtap, the kernel and uboot that shipped with ROACH boards prior to 020250. See this mailing list post for more info. Please update your ROACH according to ROACH_kernel_uboot_update page.

• A ROACH board with tgtap driver installed and tcpborphserver KATCP build after 2009-07-06 for integrated tgtap support. There have since been bugfixes. Be sure to use the latest versions, post 2009-12-22.
• A cable for loopback.
• A computer with a control interface to ROACH.
• Python KATCP and CORR libraries loaded on the control computer.
• I assume the reader is familiar with Linux and is able to use KATCP over telnet.

For basic introduction to ROACH and BORPH, try the Roach_Tutorial.

• Supplied Debian ETCH filesystem (see Getting Started with ROACH for setup details) which includes the required tcpborphserver and tgtap driver.
• If you want to compile the design from scratch, you will need a computer with the standard CASPER libraries and toolflow configured.
• A pre-configured 10GbE NIC if you want to send/receive packets from your computer.
• Matplotlib/Pylab if you want to be able to plot the transmitted and received waveforms.

• Get the model file here.
• Get the pre-compiled bitstream here.
• Get the python demo script here.
• Get the python CORR-0.4.0 library here.
• Get the python KATCP library here.
• Get the ROACH Debian Etch filesytem here. Find out how to use it on the Getting Started with ROACH page.
• Get the 10GbE userspace driver, tgtap here (included in above filesystem already).
• Get the tcpborphserver ROACH KATCP server here (included in the above filesystem already).

This demo is self-contained. Simply connect a 10GbE cable between ports 0 and 3, or plug both into a switch. 10GbE is auto MDI/MDI-X, so no crossover cable or anything special is needed.

The board clocks off an internal crystal so no external clock source is necessary.

I assume that you already have tcpborphserver and tgtap installed. tgtap should be in the path statement for tcpborphserver to use it (eg /usr/bin or /usr/local/bin). I recommend you use the pre-packaged filesystem here, which includes all the required software. See the Getting Started with ROACH guide for installation help.

Using rsync, scp, cp or NFS share, you need to have the design visible to tcpborphserver. I like scp from my linux box:

 scp r_10gbe_demo.bof root@myroach:/boffiles/

Execute the demo script like any other Linux program.

./10gbe_demo.py myroach

It will go and program the ROACH, configure the cores, setup the packet source and printout the transmitted and received data stream. You should see an output something like the following:

Connecting to server roach0135 on port 7147...  ok

------------------------ Programming FPGA... ok --------------------------- Port 0 linkup:  True Port 3 linkup:  True --------------------------- Configuring receiver core... done Configuring transmitter core... done --------------------------- Setting-up packet source... done Setting-up destination addresses... done Resetting cores and counters... done Sent 0 packets already. Received 0 packets already. ------------------------ Triggering snap captures... done Enabling output... done Reading 2048 values from bram snap_gbe0_tx_bram_msb... ok Reading 2048 values from bram snap_gbe0_tx_bram_lsb... ok Reading 2048 values from bram snap_gbe0_tx_bram_oob... ok Reading 2048 values from bram snap_gbe3_rx_bram_msb... ok Reading 2048 values from bram snap_gbe3_rx_bram_lsb... ok Reading 2048 values from bram snap_gbe3_rx_bram_oob... ok Unpacking TX packet stream...

[ 124]: data:               FD IP: 10. 0. 0. 30 [TX LED] [Link up]  [ 125]: data:               FE IP: 10. 0. 0. 30 [TX LED] [Link up]  [ 126]: data:               FF IP: 10. 0. 0. 30 [TX LED] [Link up]  [ 127]: data:              100 IP: 10. 0. 0. 30 [TX LED] [Link up] [eof]  [ 128]: data:              101 IP: 10. 0. 0. 30 [TX LED] [Link up]  [ 129]: data:              102 IP: 10. 0. 0. 30 [TX LED] [Link up]  [ 130]: data:              103 IP: 10. 0. 0. 30 [TX LED] [Link up] 

[ 892]: data:              3FD IP: 0.0.0.20 [led_rx] [led_up]  [ 893]: data:              3FE IP: 0.0.0.20 [led_rx] [led_up]  [ 894]: data:              3FF IP: 0.0.0.20 [led_rx] [led_up]  [ 895]: data:              400 IP: 0.0.0.20 [led_rx] [led_up] [eof]  [ 896]: data:              401 IP: 0.0.0.20 [led_rx] [led_up]  [ 897]: data:              402 IP: 0.0.0.20 [led_rx] [led_up]  [ 898]: data:              403 IP: 0.0.0.20 [led_rx] [led_up] 

The output is a list of every valid data word into/out of the 10GbE cores. The first column is the snap block index (valid counter), followed by the 64 bit data word, the source/destination IP address and then any flags that were set. Note that only the lower bits of the IP address are valid, because I ran out of OOB bits in the snap block for all the data.

Alternatively, you can plot this data using the -p flag on the command line. This requires MATPLOTLIB/PYLAB. You will get a picture like this:

image: R_10gbe_demo_plot_output.png

Where the top graph is the transmitted counter and the bottom graph is the received counter.

That's great, but what's going on behind the scenes, I hear you ask. The script is not performing any black magic. Here is the list of steps that it's doing for you automatically.

Open the model file in Simulink. It looks like this: image: R 10gbe demo screenshot.png Run bee_xps and start the compile. No additional libraries or special settings are needed. Compile time is about 30 minutes on my 2.4GHz laptop.

The included Python demo script uses the CORR package which wraps the generic KATCP Python package. This provides some simpler interfaces for controlling FGPAs remotely. You could SSH into ROACH and perform all these steps manually using standard Linux tools like cat, echo and hd, but this is painful and slow and so will not be covered by this tutorial. I will, however, demonstrate the KATCP control strings for those of you who haven't discovered the wonders of Python yet. To execute these commands manually, follow the steps outlined below...

• Connect to your favourite ROACH board on the KATCP control port 7147:

telnet myroach 7147

• Program the FGPA

?progdev r_10gbe_demo_2009_Jul_27_1610.bof 

• Configure the transmitter core (load the userspace driver). IP 10.0.0.20 on port 60000 with MAC 02:02:0A:0:0:14

?tap-start gbe0 02:02:0A:00:00:14 10.0.0.20 60000

• Configure the receiver core (load the userspace driver). IP 10.0.0.30 on port 60000 with MAC 02:02:0A:0:0:1E

?tap-start gbe3 02:02:0A:00:00:1E 10.0.0.30 60000

• Setup the packet source. This demo sends a 64bit counter. The rate and size of packets is configurable. Let's aim for a length of 128 64-bit words (1024 bytes, up to 8192 should be ok) every 16384 FPGA clocks.

?wordwrite pkt_sim_period 0 0x00004000 ?wordwrite pkt_sim_payload_len 0 0x00000080

• Setup where the packets should go

?wordwrite dest_ip 0 0x0A00001E ?wordwrite dest_port 0 0x0000EA60

• Reset the cores. This is important because if the TX buffer overflows, the FIFOs needs to be reset before it will resume sending.

?wordwrite gbe_core_rst 0 0x00000001 ?wordwrite gbe_core_rst 0 0x00000000

• Start transmission

?wordwrite pkt_sim_run 0 0x00000001

• Check that we're actually sending packets. Counter of packets sent so far.

?wordread gbe0_tx_cnt

• Check that we're receiving packets

?wordread gbe3_rx_frame_cnt

Then the script simply goes on to dump a snapshot of the sent and received streams.

• You can send the packets to your computer simply by changing the target IP address. Try setting the dest_ip register to your computer's 10GbE NIC's IP address and have a look at the incoming packets using Wireshark or tcpdump.
• Try increasing the packet rate and extending the script to check for lost packets (by making sure that the next received packet's counter values follow-on from the previous one). Also play with the packet sizes. Notice that you can get much higher throughputs into a computer with larger packets. You might have to enable "Jumbo frames" on your NIC's driver to receive packets larger than 1500 bytes.

• Make sure you can connect to the KATCP server

Using telnet into port 7147

 telnet myroach 7147

You should be greeted by a screen similar to:

Trying 192.168.1.10... Connected to myroach.mydomain.net. Escape character is '^]'. #version tcpborphserver-1.0 #build-state tcpborphserver-0. #log info 1248737709081 tcpborphserver new_connection_192.168.1.101:59876_to_192.168.1.10:7147

• Check that you have the required tgtap command available by requesting a list of available commands:

?help

You should get a list of escaped commands. Look for:

#help tap-stop stop_the_tap_server_(?tap-stop) #help tap-start start_the_tap_server_(?tap-start_10gbe-register-name_mac-address_ip-address_fabric-port)

If these commands are not listed, you need a new version of tcpborphserver.

• Check that you can program the FPGA

?progdev r_10gbe_demo.bof

You should get:

!progdev ok

If this failed, likely you didn't copy the file correctly. Make sure it has its executable flag set.

• Check that both cables are plugged-in by looking at the "linkup" lines of the demo script, or reading the registers manually over KATCP:

?wordread gbe0_linkup

and

?wordread gbe3_linkup

In both cases, you should get

!wordread ok 0x1

If a zero value is returned, the cable is disconnected.

• Check that the 10GbE cores were correctly configured. Run the demo script with the -a flag to see the ARP table (make sure entries for 10.0.0.20 and 10.0.0.30 have been set correctly) and print out useful core debug info.

============================ 10GbE Receiver core details: ============================ ------------------------ GBE0 Configuration... My MAC:  02 02 0A 00 00 1E  Gateway:    0   0   0  30  This IP:   10   0   0  30  Gateware Port:  60000 Fabric interface is currently:  Enabled XAUI Status:   0 1EA60 XAUI PHY config:     RX_eq_mix:  0    RX_eq_pol:  1    TX_pre-emph: EA    TX_diff_ctrl: 60 ARP Table:                   IP:  10.  0.  0. 17: MAC: FF FF FF FF FF FF  IP:  10.  0.  0. 18: MAC: FF FF FF FF FF FF  IP:  10.  0.  0. 19: MAC: FF FF FF FF FF FF  IP:  10.  0.  0. 20: MAC: 02 02 0A 00 00 14  IP:  10.  0.  0. 21: MAC: FF FF FF FF FF FF  IP:  10.  0.  0. 22: MAC: FF FF FF FF FF FF  IP:  10.  0.  0. 23: MAC: FF FF FF FF FF FF  IP:  10.  0.  0. 24: MAC: FF FF FF FF FF FF  IP:  10.  0.  0. 25: MAC: FF FF FF FF FF FF  IP:  10.  0.  0. 26: MAC: FF FF FF FF FF FF  IP:  10.  0.  0. 27: MAC: FF FF FF FF FF FF  IP:  10.  0.  0. 28: MAC: FF FF FF FF FF FF  IP:  10.  0.  0. 29: MAC: FF FF FF FF FF FF  IP:  10.  0.  0. 30: MAC: 02 02 0A 00 00 1E  IP:  10.  0.  0. 31: MAC: FF FF FF FF FF FF