Notes on server and FPGA configuration - IAA-BURSTT/document GitHub Wiki

This page will introduce the concept of the ZCU216 RFSOC board (hereafter FPGA board) and Server configurations. The first step in the configuration is deciding the server (100G NIC) IP addresses (dest_IP). This depends on whether you want to perform bf64 or bf256 operations. The bf16 mode can piggyback on either of them. After the server dest_IP is set, we need to update the ARP table, which will be written to the FPGA ether core. Then an FPGA config file is required per FPGA board per operation mode. An overview is summarized in the figure below:

The FPGA board is controlled through the ethernet port. A DHCP server will provide the control_IP addresses to all FPGAs connected to the same network switch. Upon receiving the new FPGA board, a custom-defined MAC address is written onto the board. A lookup table (/etc/dnsmasq.conf) is used to cement the control_IP of each board. All servers connected to this switch can talk to and control the FPGA boards. However, only one server should act as the DHCP server. The dnsmasq service should be enabled and started on this server. If more than one server on this switch runs the dnsmasq service, the FPGA control may become unstable.

To enable the service (only need to do once):

sudo systemctl enable dnsmasq

To check the status:

sudo systemctl status dnsmasq

To start/restart the service:

sudo systemctl restart dnsmasq

In Fushan, 'burstt9' is acting as the DHCP server, while 'burstt1', 'burstt2', 'burstt3', and 'burstt4' are also connected to the control switch.

In Nantou, 'burstt5' is the only server and therefore serves the DHCP.

In IAA, 'burstt8' is currently acting as the DHCP server.

One needs to decide the IP addresses based on the intended operation mode. Please refer to the summary diagram above.

For the current BURSTT servers (burstt1 -- burstt9), the two sets of dual-port 100G NICs conventionally have the names: ens2f0np0, ens2f1np1, ens4f0np0 and ens4f1np1 (ens2 and ens4 reflect the PCIe interface used). f0np0 and f1np1 are the two ports of one NIC. Note that ens2 is associated with CPU0 and NuMA node0, while ens4 is associated with CPU1 and NUMA node1. Each NUMA node can be thought of as a mini-server that processes the data received by one NIC (both f0np0 and f1np1 ports). Therefore each node processes half the spectrum from all 4 FPGAs in bf64 mode, or 1/8 of the spectrum from all 16 FPGAs in the bf256 mode.

To set the 100G NIC IP addresses, there are several ways:

1. use '/opt/burstt/bin/set_nic_ip.sh' edit the IP in the file and run the script (both needs sudo).
2. use 'nmtui' Choose 'Edit a connection' and modify each interface. Besides the IP setting, also check the section to make sure MTU is 9216. Remember to check 'never as default route'. Save the settings (hit ) and navigate back to the starting menu. Then choose 'Activate a connection' and deactivate / activate each interface (ens2 and ens4 interfaces).

Some notes on the Server 100G NIC IP (dest_IP). For bf64 and bf256, the spectrum is split up into several packets and each packet will go to a different dest_IP. Their packet_dest_IP = dest_IP0 + packet_order, where dest_IP0 is given by our fpga_config. Internally, the last 2 bits (for bf64) or 3 bits (for bf256) of the dest_IP0 is masked. So a .5 or .6 IP0 is equivalent to .4 in bf64. Similarly, a .15 IP0 is equivalent to .8 in the bf256 case. Then packet_order is added onto this masked IP0 for the actual destination IP.

Consider two servers on the same 100G switch, both working in the bf64 mode. If we set the IP of server1 as .40 .41 and set .42 .43 for server2, then for the fpga_configs, we would set IP0 to .40 for server1. However, if we set IP0 to .42 for FPGAs intended for server2, their packets will actually still go to server1 (.40 and .41). To avoid this conflict, we need to set the IP of server2 to .44 .45 instead (at least +4 from server1), and the IP0 to .44. This is reflected in the dest_IP table in the summary diagram at the beginning of this page.

A second note is that we commonly reserve .8 to .23 for FPGA local_IPs (up to 16 FPGAs). Therefore any IPs above this range can be chosen for the server NIC IP. For example, burstt14 is on a separate 100G switch, so it can use the following IPs (for bf64):

ens2f0np0 10.17.16.24

ens4f0np0 10.17.16.25

ens2f1np1 10.17.16.28

ens4f1np1 10.17.16.29

Any server on the 'control switch' can act as the control server for the FPGAs. To do so, the server should have the CASPER RFSOC python module installed. Also, it should contain the bitcode library for our FPGA. In outrigger station with a single server, it will naturally be the control server (e.g. burstt5 in Nantou). For sites with multiple servers, it is commonly the same server that serves the DHCP (dnsmasq). In IAA, this is burstt8. In Fushan, this is burstt9. However, other servers can install these module and library to control the FPGAs.

To activate the python environment where the module is installed, type the following in a terminal:

rfsoc

This will activate the (rfsoc) environment and change to the /data/rfsoc/python_zcu216/ directory. The bitcode library can be found under /data/rfsoc/.

The fpga_configs/ subfolder in the working directory (i.e. /data/rfsoc/python_zcu216/fpga_configs/) contains config files for various sites and configurations. In each config file, there are two sections. The [board] section is the specific setting of IPs and should be different for each FPGA board. The MAC address in this section is an arbitrary setting we give to the 100G interface on the FPGA (corresponding to the local_IP). Therefore, we conventionally just copy the IPv4 value and write them in HEX as the MAC. This MAC address is not critical because there is no return traffic to the FPGA on the 100G interface.

The [common] section is the same for all FPGA boards, under the same operation mode (i.e. 16bit, bf16, bf64 or bf256). Each mode will have a different bitcode as well as some other different settings. It is in general OK to copy the [common] section of the same mode and edit just the [board] section when creating a new fpga config file.

An important part of the [common] section (besides choosing the bitcode, i.e. the fpgfile) is specifying the ARP table. This will be described below.

The 100G switch uses the destination MAC address in each UDP packet we send from the FPGA to determine where the packet will be directed. In a normal network condition, the switch will automatically gather the MAC address from connected devices and broadcast it back to the sender so that the correct MAC can be set in the packet header. However, this mechanism for some reason does not work with the FPGA 100G interface. Or else we did not find the right setting. Alternatively, the FPGA supports a static ARP table that translates the dest_IP to dest_MAC.

The ARP table is stored in the YAML config file format under the config_arp/ subfolder. The ARP table we will use is then specified in the fpga_config file. The most important information in the ARP table (YAML file) is thus the server 100G NIC IP and its MAC address. For completeness, we also store the FPGA local_IP and the corresponding MAC address that we arbitrarily assigned (i.e. HEX version of the local_IP) in the ARP table.

In addition to the ARP table to be written to the FPGAs, we also need to specify a static MAC on the 100G switch for ports connected to the server NIC.

In the B3 server room, we have 20 FPGAs and several servers. The roles of the servers are:

• DHCP (dnsmasq) server: burstt8
• rfsoc server: burstt8
• beamforming (rudp) server: burstt6 (in this example)

First, we need to find out the 100G NIC's IP and MAC addresses from burstt6. They are

• ens2f0np0 = 10.17.16.26 88:e9:a4:97:ba:56
• ens2f1np1 = 10.17.16.34 88:e9:a4:97:ba:57
• ens4f0np0 = 10.17.16.27 88:e9:a4:97:ba:2e
• ens4f0np0 = 10.17.16.35 88:e9:a4:97:ba:2f

Second, we can edit the ARP table on the DHCP server at burstt8:~/rfsoc/python_zcu216/config_arp/config.yaml.SW.14APR24

The filename is arbitrary but the correct name should be used in the FPGA config file later.

In the ARP table above, we have already assigned the localip and mac address for the FPGAs (in the f-engine section). We now need to edit the FPGA configs (at burstt8:~/rfsoc/python_zcu216/fpga_configs/b3sw256/fpga10?_bf16.config). The [common] section is the same for all 4 files. The [board] section is different for each file and is listed here:

In fact, a new version of script (zcu216_100g_config2_log.py) has two modes of loading the FPGA configs. The first mode is the same as the original one, where we use '-c <fpga_config_file>'. The <fpga_config_file> should contain all the settings. The second mode can accept two config files with '-c <board_config>' and '-C <common_config>'. Here '-c' (lower case) takes the config file with only the board section while '-C' (capital) takes the config file with only the common section. With this separation, it would be easier to maintain the common section.

Start two consoles and enter the (rfsoc) environment from both.

In the first console (in burstt8:~/rfsoc/python_zcu216/), there is a script: reload_burstt10.py. Execute this file and supply 'bf16' as the mandatory argument. This is because the FPGAs are now connected to the NICs without going through the SW. So both packets should be sent to the same dest_ip. This is what the 'bf16' bitcode does.

./reload_burstt10.py bf16

Alternatively, there is also a more general 'reload_fpga.py' script that can reload an arbitrary list of FPGAs with selected configs. Please feel free to explore it.

In the second console, go to the config_arp subfolder and start the interactive python:

cd config_arp
python -i zcu_newcontrol.py
> f6 = connect(['100','101','102','103'])
> resetepoch(f6)
> fpgaIO('r', f6, 'clk_frequency', show_key=True)
> fpgaIO('r', f6, 'epoch_second')

The 'resetepoch' is necessary to align the packet counters. The other two 'fpgaIO' commands are utility to check whether things are working properly.