10GbE_and_IB_Networking - david-macmahon/wiki_convert_test GitHub Wiki

Here are some results of some network bandwidth testing done at Swinburne. The aim of the testing was to see how fast we could capture UDP packets on a 10GbE PCIe card. We also performed somes tests to see how fast data could be streamed between 2 servers and written to a GPU on the receiving server.

ROACH to Server using UDP + 10GbE

A simple ROACH design was created that generates 8KB UDP packets at a configurable data rate. The UDP packets are just packed with a 1024 x 64-bit unsigned integers that correspond to the packet sequence number. The first 10GbE port on the ROACH was directly cabled to a 10GbE PCIe Card (Myricom 10G-PCIE-8A). A simple UDP capture program, written using the PSRDADA library, was used to read packets from the socket and write them to a shared memory ring buffer. The machine used for the test was an Intel Server with 4 x X7560 CPUs. The Interrupt Coalsesing for the RX queue (rx-usecs) was left at the default of 75 usecs. The capture program was bound to a core on the second CPU as the first CPU seemed to be rather busy processing interrupts, etc. The highest rate sustained rate with no packet loss (for 120s) was ~1008 MB/s (7.88 Gb/s). In this instance the capture process consumed ~92% of 1 core.

Server -> IB -> Server -> GPU

In this benchmark, we tested the Infiniband and GPU bandwidth in a streaming scenario. Each of the 2 servers used had the following specifications:

  • BOARD: Tylersburg IOH and the ICH10
  • NIC: QLogic QDR IB (Infiniband) QLE7340
  • GPU: Nvidia C2070 (Fermi)
  • CPU: Dual Intel Xeon X5650's (2.66 GHz, 6 core, 12ML3)
  • RAM: 48 GB DDR3 (1333 MHz)

The two servers were connected together via 40Gbps QDR IB via an IB switch. The details of the tests were:

Server A:

  • Process A: generating a data stream, writing it to shared memory ring buffer
  • Process B: reading from shared memory ring buffer and writing via RDMA RC (Reliable Connection) IB Verbs to Server B

Server B:

  • Process A: receives the RDMA RC data stream from Server A, writing the data into shared memory ring buffer
  • Process B: reading data from shared memory ring buffer, and transferring (cudaAsyncMemcpy) to GPU

The sustainable rate for this was 3215 MB/s (25.1 Gb/s)