This manual describes how to set up the NetFPGA-10G board and run the included Production Test software. The Production Test verifies that your board is working correctly. You should run the Production Test before attempting any development with NetFPGA-10G. You should also run the RLDRAM Test. If either test indicates a problem with your board, return it immediately to HiTech Global (HTG).

The Production Test is performed by the NetFPGA-10G’s Virtex5 FPGA. The FPGA design simultaneously exercises all of the board-level interfaces, with the exception of the RLDRAM. The status of the tests is displayed on the board’s LEDs and can also be read back either via PCIe or serially using the UART. A bitstream containing the Production Test FPGA design (along with supporting software) is provided in the NetFPGA-10G github repository in the projects/production_test/ subdirectory.

In order to download bitstreams to the FPGA and CPLD devices on the NetFPGA-10G board, you will need to install the Xilinx ISE software and the associated JTAG programming cable drivers. ISE can be downloaded from Xilinx's Download site at www.xilinx.com/support/downloads. Installation instructions for supported Windows and Linux platforms can be found at Xilinx's Product Support & Documentation site, www.xilinx.com/support.

The NetFPGA-10G Production Test can be run in one of two modes. In Server Mode, the NetFPGA-10G board is installed as PCIe expansion card in a host PC, allowing most major on-board systems to be verified. In Standalone Mode, the board operates without a PCIe host. In this mode, the PCIe interface is not tested.

It is highly recommended that the Production Test be run in Server Mode.

The following items are required to conduct the test:

• NetFPGA-10G board – revision 4 or higher

• Production Test project – stored in the projects/production_test/ subdirectory of the NetFPGA-10G git repository

• Linux host PC:

The NetFPGA-10G board has been tested successfully in host PCs with the following hardware configurations**:**

Processor

Intel Core i7 920 or Intel Core i7 960

Motherboard

eVGA X58 141-BL-E758 or eVGA 132-GT-E768

Memory 6GB (3x2GB) DDR3 modules from Corsair

Power Supply

500W PSU from Antec

A NetFPGA-10G Prebuilt Desktop 10G configuration can be ordered from Accent Technology.

Note: The NetFPGA-10G board is not compatible with the Intel DX58S02 motherboard.

The Production Test has been run successfully on Fedora 14 (x86_64) with Linux kernel versions 2.6.35.14-95.fc14.x86_64, 2.6.35.13-92.fc14.x86_64 and 2.6.35.6-45.fc14.x86_64. The NetFPGA team has adopted Fedora 14 (x86_64) as the reference operating system for deployment. If you wish to use Fedora 14 (x86_64) for your development work, follow the instructions on the Reference Operating System page.

• Xilinx JTAG programming cable

• An RS232 serial (null-modem) cable

• Serial Terminal Emulation software to communicate with the NetFPGA-10G board over the RS232 interface:

For Windows, there are a number of freely-available Terminal Emulators, including PuTTY. On Linux, use minicom. Fedora's built-in package manager should allow you to install minicom from the GUI using Add/Remove Software. If not, RPMs for minicom (and lrzsz, on which it depends) can be found at pkgs.org.

To test the 10GE and Expansion Interfaces, you will also need these cables which create loopbacks on the NetFPGA-10G’s expansion connector and 10GE ports. These are optional.

• A high-speed twinax cable assembly for the Expansion Interface – part number HQDP-020-05.00-STL-SBR-2; available from Samtec
• A pair of 10GE SFP+ cables – the following cables have so far been tested and verified: Verified 1G and 10G cables

Before running the Production Test, you need to set up your NetFPGA-10G board and install it in the host PC.It is CRITICAL you follow the directions closely; in particular you must provide a power source on the ATX connector regardless of the test. The board requires power from the ATX power source at all times. Follow these steps:

• Ensure Jumper J16 is not populated. This jumper is located at the top left corner of the board, underneath the RS232 DB-9 connector, as shown below.
  

• Set switch SW9 to _PCI _to power the board from the PCIe supply. Make sure the switch is set fully in the PCI position as shown below.
  

• Ensure the DIP switches SW1, SW3, _SW6 _and _SW10 _are set correctly, matching the figure below.

SW1: SEL0, SEL1, M2, M1, M0, N2, N1, N0 are set to off, off, off, on, on, on, off, off
SW6: SEL0, SEL1, M2, M1, M0, N2, N1, N0 are set to off, off, off, on, off on, off, on
SW3: M0, M1, M2 is set to on, on, on
SW10: SEL0, SEL1, SEL2 are set to on, off, off

• Power down the host PC and install the NetFPGA-10G card securely in a free PCIe slot

• Connect one of the host PC's available ATX power cables to the ATX supply connector on the NetFPGA-10G board.
  

• If you plan to test the NetFPGA-10G board Expansion Interface, connect a Samtec twinax cable to the Expansion Interface to form a loop, as shown below:
  

• If you plan to test the 10GE interface, connect a pair of 10GE SFP+ cables to the board's 10GE interface. Each cable interconnects two adjacent 10GE ports, forming loops as shown below:
  

• Connect an RS232 serial cable to the RS232 port on the NetFPGA-10G board. Connect the other end to your PC's COM (Serial) port, or to a suitable adapter.
  

• Connect a Xilinx JTAG programming cable to the JTAG connector on the NetFPGA-10G board. Connect the programming cable's USB or parallel port cable to the computer which will be used to program the on-board devices.
  

Your board should now be installed in the host PC and connected as shown in the diagram below.

The photograph below shows a NetFPGA-10G card installed in Server Mode within a host PC.

To run the Production Test on the host PC, follow these steps:

1. Boot the host PC. As root, open the file /boot/grub/grub.conf in a text editor and follow these instructions:

1. Locate the entry for your particular kernel (the default=N says that your default kernel is the Nth entry).

2. Find the line starting with kernel.

3. Append memmap=256M$0x5f700000 to the end of that line, so that your grub.conf file now looks similar to the example shown below. The memmap=256M$0x5f700000 parameter is passed to the kernel at boot time. It requests that a 256MB region of memory starting at the address 0x5f700000 be reserved. This block will be used later in the NetFPGA-10G PCIe DMA test.  

4. Save grub.conf.

2. Start Xilinx iMPACT. If necessary, create a new project. Connect to the JTAG programming cable. You should see the Virtex5 FPGA and the CPLD on the programming cable's JTAG chain, as shown below.
   
   Program the CPLD with the design located at production_test/bitfiles/cpld.jed. Program the FPGA with the bitstream located at production_test/bitfiles/prod_test_no_rldram.bit. iMPACT may ask for attaching a PROM device, select 'no'. Programming the devices takes a few seconds. (For instructions on how to install and configure the Xilinx software tools and a driver for the JTAG programming cable, please refer to "Xilinx Software Installation" earlier in this document.)

3. Reboot the host PC.

4. Verify that the DMA memory block has been reserved correctly by following these steps:

1. From a shell, query the kernel's boot command line using:  

cat /proc/cmdline

2. You should see something like:  

If you don't see memmap=256M$0x5f700000 at the end of these parameters, go back to step 1 to ensure you correctly edited grub.conf

5. Verify that the NetFPGA-10G board is installed correctly by following these steps: 1. From a shell, search for the NetFPGA-10G board on the host PC's PCIe bus using:

lspci –vxx | grep Xilinx –A3

(If the lspci command is not found, then it may not be in your path. Try /sbin/lspci)

2. You should see something like the screenshot below (although the memory addresses and IRQ may differ):  

If not, please check that the board is properly installed and connected to the host PC's ATX power supply. Ensure that the on-board FPGA and CPLD have been correctly programmed.

6. Check the status LEDs on the NetFPGA-10G board by following these steps: 1. The POWER LEDs should all be illuminated, as shown here:
   2. FPGA LED0 is a heartbeat signal derived from the system clock. If this LED is not flashing, then there is no need to continue any further as it is highly unlikely that the test will run. Go back to Step 1.

3. FPGA LED1 illuminates to indicate that the Expansion Interface loopback test has passed. (This requires the samtec cable to be fitted.)

4. FPGA LED2 illuminates to indicate that all tests, with the exception of the Expansion Interface test, have passed. This includes tests on the SRAM, all 10GE interfaces, the CPLD and platform flash interface, power supplies and clocks. The PCIe and RS232 interfaces must be tested separately (see below).

5. CPLD LEDs 0 and 2 should be illuminated to indicate that the PCI reset and flash device are functional. CPLD LED1 illuminates to indicate that the FPGA is successfully programmed. Finally, CPLD LED3 flashes when the clock to the CPLD is active and all power levels are correct.  

7. On the host PC, run the Production Test by following these steps:

1. In a shell, go to the root of your NetFPGA-10G repository. Enter the Production Test project sw/ subdirectory using:  

cd projects/production_test/sw/

2. Build the test using:  

make

3. Enter the scripts subdirectory and run the Production Test software using:  

cd scripts
sudo ./production_test.py

8. The Production Test output should look something like this:
   
   If you would like the test to generate a report in the output/reports/ directory, you can provide the test script with a filename argument:

sudo ./production_test.py report001.txt

(Note that you will not see test output on stdout in this case.)

The RS232/UART Interface Test results indicate that the host successfully communicated with the NetFPGA-10G board over the UART. The host PC handshakes with the MicroBlaze processor on the FPGA via the UART using a predetermined test sequence. The PCIe DMA test checks communication with the board over the PCIe interface using the block of memory reserved earlier. Data is streamed over PCIe to the FPGA, which then returns the data to the host for checking. The DMA test reports the total volume of data transferred over the PCIe interface during a ten-second period. Finally, the FPGA System Report shows the results of tests on the other major on-board systems, excluding the RLDRAM. Test results are read from registers in the FPGA design and compared to reference results to determine PASS or FAIL. Tests on the 10GE and Expansion (Samtec) Interface can be expected to fail if you have not connected the optional cables to these ports. Failures on these tests do not affect any other system test results.

Note that if the 10GE interface test fails (because of a missing cable, for example), this FAIL result will persist until the board is reset or re-programmed, even if the cable is later reconnected.

The following items are required to conduct the test:

• NetFPGA-10G board – revision 4 or higher

• Production Test project – stored in the projects/production_test/ subdirectory of the NetFPGA-10G git repository

• ATX PC power supply (or equivalent)

• Xilinx JTAG programming cable

• An RS232 serial (null-modem) cable

• Serial Terminal Emulation software to communicate with the NetFPGA-10G board over the RS232 interface:

For Windows, there are a number of freely-available Terminal Emulators, including PuTTY. On Linux, use minicom. Fedora's built-in package manager should allow you to install minicom from the GUI using Add/Remove Software. If not, RPMs for minicom (and lrzsz, on which it depends) can be found at pkgs.org.

To test the 10GE and Expansion Interfaces, you will also need these cables to create loopbacks on the NetFPGA-10G’s expansion connector and 10GE ports. These are optional.

• A high-speed twinax cable assembly for the Expansion Interface – part number HQDP-020-05.00-STL-SBR-2; available from Samtec
• A pair of 10GE SFP+ cables – the following cables have so far been tested and verified: Verified 1G and 10G cables

Before running the Production Test, follow these steps to set up your board:

• Place the board on a flat surface and disconnect all attached connections.

• Ensure Jumper J16 is not populated. This jumper is located at the top left corner of the board, underneath the RS232 DB-9 connector, as shown below.
  

• Set switch SW9 to _ATX _to power the board from an external supply. Make sure the switch is set fully in the _ATX _position as shown below.
  

• Ensure the DIP switches SW1, SW3, _SW6 _and _SW10 _are set correctly, matching the figure below.

SW1: SEL0, SEL1, M2, M1, M0, N2, N1, N0 are set to off, off, off, on, on, on, off, off
SW6: SEL0, SEL1, M2, M1, M0, N2, N1, N0 are set to off, off, off, on, off on, off, on
SW3: M0, M1, M2 is set to on, on, on
SW10: SEL0, SEL1, SEL2 are set to on, off, off

• Connect an ATX power cable to the ATX supply connector on the NetFPGA-10G board.
  

• If you plan to test the NetFPGA-10G board Expansion Interface, connect a Samtec twinax cable to the Expansion Interface to form a loop, as shown below:
  

• If you plan to test the 10GE interface, connect a pair of 10GE SFP+ cables to the board's 10GE interface. Each cable interconnects two adjacent 10GE ports, forming loops as shown below:
  

• Connect an RS232 serial cable to the RS232 port on the NetFPGA-10G board. Connect the other end to your PC's COM (Serial) port, or to a suitable adapter.
  

• Connect a Xilinx JTAG programming cable to the JTAG connector on the NetFPGA-10G board. Connect the programming cable's USB or parallel port cable to the computer which will be used to program the on-board devices.
  

Your board is now ready for testing and should be connected as shown in the diagram below.

To run the Production Test, follow these steps:

1. Apply power to the board and use the POWER LEDs to check the board's power rails. All power LEDs should be illuminated steadily and equally as shown below.

2. Start Xilinx iMPACT. If necessary, create a new project. Connect to the JTAG programming cable. You should see the Virtex5 FPGA and the CPLD on the programming cable's JTAG chain, as shown below.

Program the CPLD with the design located at production_test/bitfiles/cpld.jed. Program the FPGA with the bitstream located at production_test/bitfiles/prod_test_no_rldram.bit. iMPACT may ask for attaching a PROM device, select 'no'. Programming the devices takes a few seconds. (For instructions on how to install and configure the Xilinx software tools and a driver for the JTAG programming cable, please refer to "Xilinx Software Installation" earlier in this document.) 3. Check the CPLD and FPGA status LEDs on the NetFPGA-10G board by following these steps: 1. FPGA LED0 is a heartbeat signal derived from the system clock. If this LED is not flashing, then there is no need to continue any further as it is highly unlikely that the test will run. Go back to Step 1.

2. FPGA LED1 illuminates to indicate that the Expansion Interface loopback test has passed. Note that this requires the samtec cable to be fitted.

3. FPGA LED2 illuminates to indicate that all tests, with the exception of the Expansion Interface test, have passed. This includes tests on the SRAM, all 10GE interfaces, the CPLD and platform flash interface, power supplies and clocks. The PCIe and RS232 interfaces must be tested separately (see below).

4. CPLD LEDs 0 and 2 should be illuminated to indicate that the PCI reset and flash device are functional. CPLD LED1 illuminates to indicate that the FPGA is successfully programmed. Finally, CPLD LED3 flashes when the clock to the CPLD is active and all power levels are correct.  

4. Start a serial terminal session on the PC using PuTTY, _minicom _or equivalent. Connect to the NetFPGA-10G board at 9600bps with 8 data bits, no parity bit, 1 stop bit and no flow control. Connecting to the board via the UART provides more detailed test results than those shown on the CPLD and FPGA LEDs. It also allows the user to control the tests interactively.
5. Check your serial terminal window. You may need to press a key before you see the NetFPGA-10G Test prompt (highlighted in the screenshot below). Follow these steps: 1. Press "i" to initialize the test. This resets all test structures inside the FPGA, along with peripheral devices and the 10G PHYs. 2. When the prompt returns, press "s" to start the test. After a few seconds, your serial terminal should look like this:
   

The screenshot above shows the results of tests on almost all of the major on-board systems, excluding the RLDRAM. Test results are read from registers in the FPGA design and compared to reference results to determine PASS ("OK") or FAIL. Tests on the 10GE and Expansion Interfaces can be expected to fail if you have not connected the optional cables to these ports. Failures on these tests do not affect any other system test results.

Note that if the 10GE interface test fails (because of a missing cable, for example), this FAIL result will persist until the board is reset or re-programmed, even if the cable is later reconnected. Note also that Standalone Mode does not test the NetFPGA-10G board's PCIe interface.

For more details on the various interface tests please refer to the NetFPGA-10G Production Test project page. This page describes in detail how the Production Test exercises each of the major NetFPGA-10G system components. The page also lists some known shortcomings in the current version of the test design.

Before you start development on the NetFPGA-10G board, remember to run the separate RLDRAM Test.