Linux Network Tuning - shawfdong/hyades GitHub Wiki

ESnet (Energy Sciences Network) maintains excellent guides on how to tune Linux, Mac OSX, and FreeBSD hosts for maximum network performance, at speeds of 1Gbps or higher. We've applied their guidelines to our Linux hosts.

Table of Contents

Default Settings

To read kernel parameters on Linux, one run, e.g.:

# sysctl net.core.wmem_max
net.core.wmem_max = 229376

The following are the default setting on RHEL 6.4:

net.core.wmem_max = 229376
net.core.rmem_max = 229376
net.ipv4.tcp_wmem = 4096	16384	4194304
net.ipv4.tcp_rmem = 4096	87380	4194304
net.core.netdev_max_backlog = 100
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_mtu_probing = 0

The default Transmit Queue Length (txqueuelen) is 1000:

# ifconfig em2 | grep txqueuelen
          collisions:0 txqueuelen:1000 

TCP Tuning

We adopt the following parameters, which are recommended for a Linux host with 10G NIC, optimized for network paths up to 200ms RTT:[1]

# http://fasterdata.es.net/host-tuning/linux/
# allow testing with buffers up to 128MB
net.core.rmem_max = 134217728 
net.core.wmem_max = 134217728 
# increase Linux autotuning TCP buffer limit to 64MB
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
# increase the length of the processor input queue
net.core.netdev_max_backlog = 250000
# recommended default congestion control is htcp 
net.ipv4.tcp_congestion_control=htcp
# recommended for hosts with jumbo frames enabled
net.ipv4.tcp_mtu_probing=1

The above lines are appended to /etc/sysctl.conf to be persistent.

Note to modify a kernel parameter at runtime, we run, e.g.:

sysctl -w net.ipv4.tcp_rmem="4096 87380 67108864"
Note that white spaces are significant!

Or we can collect those parameters in a file and modify them in a batch. For example:

# cat /tmp/tunetcp.conf

# http://fasterdata.es.net/host-tuning/linux/
# allow testing with buffers up to 128MB
net.core.rmem_max = 134217728 
net.core.wmem_max = 134217728 
# increase Linux autotuning TCP buffer limit to 64MB
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
# increase the length of the processor input queue
net.core.netdev_max_backlog = 250000
# recommended default congestion control is htcp 
net.ipv4.tcp_congestion_control=htcp
# recommended for hosts with jumbo frames enabled
net.ipv4.tcp_mtu_probing=1

# sysctl -p /tmp/tunetcp.conf

txqueuelen

Increase Transmit Queue Length for 10G NICs with:

/sbin/ifconfig em1 txqueuelen 10000
/sbin/ifconfig em2 txqueuelen 10000

To make the change permanent, also append the above 2 lines to /etc/rc.d/rc.local.

Note ifconfig is obsolete. The replacement is ip:[2]

/sbin/ip link set em1 txqueuelen 10000
/sbin/ip link set em1 txqueuelen 10000

To make the change permanent, instead of modifying /etc/rc.d/rc.local, we can create a udev rules file /etc/udev/rules.d/75-custom-txqueuelen.rules:

#
# custom txqueuelen parameter value
#
KERNEL=="em[1,2]", RUN+="/sbin/ip link set %k txqueuelen 10000"

To apply the change on the fly, run:

/sbin/udevadm trigger

References

  1. ^ Linux Tuning
  2. ^ Persistent txqueuelen network parameter using udev
⚠️ **GitHub.com Fallback** ⚠️