VEGAS_OS_Modifications - david-macmahon/wiki_convert_test GitHub Wiki

Kernel UDP Buffer Modifications

The following changes must be made to /etc/sysctl.conf to increase the kernel UDP buffer size:

Existing lines:

 net.core.rmem_max = 16777216  net.core.rmem_default = 8388608  net.core.netdev.max_backlog = 3000

Replace with:

 net.core.rmem_max = 2147483648  net.core.rmem_default = 536870912  net.core.netdev.max_backlog = 50000

Also add the following line:

 kernel.sem = 1024 32000 32 32767

NVIDIA Driver

In some cases, the NVIDIA driver does not load properly and create the device files. The following two line /etc/modprobe.d/nvidia.conf file should be created, to create the NVIDIA device files automatically when the nvidia module is loaded

 install nvidia /sbin/modprobe --ignore-install nvidia $CMDLINE_OPTS && mknod -m 666 /dev/nvidia0 c 195 0 && mknod -m 666 /dev/nvidiactl c 195 255  remove nvidia /sbin/modprobe -r --ignore-remove nvidia && rm -f /dev/nvidia0 /dev/nvidiactl

IRQ Processor Affinity

System throughput is often improved by isolating the handling of high volume interrupt requests (IRQs) to a single processor in an SMP host. This isolation is called processor affinity. It is particularly important to do this if you have created processor affinity for any or all of the VEGAS data acquisition threads. For one of these threads to be removed from a processor by an IRQ, with nowhere else to execute, can degrade system throughput. A common system modification for running VEGAS is to create a processor affinity for the ethernet interface that communicates with the ROACH board, as this interface is typically responsible for a high volume of IRQs. This is the example used here.

First we need to know the IRQ number. Suppose that the ROACH facing ethernet port is eth2. In order to determine the IRQ number for eth2 run:

   cat /proc/interrupts | grep eth2

The first column in the output is the IRQ number. On the reference system, this was IRQ 83. To determine the current processor affinity (if any) for IRQ 83:

   cat /proc/irq/83/smp_affinity

The output indicates current affinity as a boolean bit field. It is given in hexadecimal, with the rightmost digit being the least significant, ie the booleans for the 4 lowest numbered processors. A binary 1 means that the processor is enabled to handle this IRQ, while a 0 means that it is not. If the low order 4 hexadecimal digits on a 16 processor machine reads:

   ffff

then all processors are enabled to handle this IRQ, ie there is no affinity. If we want to isolate this IRQ to cpu15, for example, we would:

   echo 8000 > /proc/irq/83/smp_affinity    cat /proc/irq/83/smp_affinity

The low order digits should now read:

   8000

At this point, IRQ 83 will be handled only by CPU 15. This will take effect with the next interrupt.

Note that this will not survive a reboot. In order to have processor affinity set as you would like it on the next boot, place the echo command in a boot script.

It is recommended that you disable irqbalance if you are setting your own processor affinity for IRQs.