Linux Network Tuning - shawfdong/hyades GitHub Wiki
ESnet (Energy Sciences Network) maintains excellent guides on how to tune Linux, Mac OSX, and FreeBSD hosts for maximum network performance, at speeds of 1Gbps or higher. We've applied their guidelines to our Linux hosts.
To read kernel parameters on Linux, one run, e.g.:
# sysctl net.core.wmem_max net.core.wmem_max = 229376
The following are the default setting on RHEL 6.4:
net.core.wmem_max = 229376 net.core.rmem_max = 229376 net.ipv4.tcp_wmem = 4096 16384 4194304 net.ipv4.tcp_rmem = 4096 87380 4194304 net.core.netdev_max_backlog = 100 net.ipv4.tcp_congestion_control = cubic net.ipv4.tcp_mtu_probing = 0
The default Transmit Queue Length (txqueuelen) is 1000:
# ifconfig em2 | grep txqueuelen
collisions:0 txqueuelen:1000
We adopt the following parameters, which are recommended for a Linux host with 10G NIC, optimized for network paths up to 200ms RTT:[1]
# http://fasterdata.es.net/host-tuning/linux/ # allow testing with buffers up to 128MB net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 # increase Linux autotuning TCP buffer limit to 64MB net.ipv4.tcp_rmem = 4096 87380 67108864 net.ipv4.tcp_wmem = 4096 65536 67108864 # increase the length of the processor input queue net.core.netdev_max_backlog = 250000 # recommended default congestion control is htcp net.ipv4.tcp_congestion_control=htcp # recommended for hosts with jumbo frames enabled net.ipv4.tcp_mtu_probing=1
The above lines are appended to /etc/sysctl.conf to be persistent.
Note to modify a kernel parameter at runtime, we run, e.g.:
sysctl -w net.ipv4.tcp_rmem="4096 87380 67108864"Note that white spaces are significant!
Or we can collect those parameters in a file and modify them in a batch. For example:
# cat /tmp/tunetcp.conf # http://fasterdata.es.net/host-tuning/linux/ # allow testing with buffers up to 128MB net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 # increase Linux autotuning TCP buffer limit to 64MB net.ipv4.tcp_rmem = 4096 87380 67108864 net.ipv4.tcp_wmem = 4096 65536 67108864 # increase the length of the processor input queue net.core.netdev_max_backlog = 250000 # recommended default congestion control is htcp net.ipv4.tcp_congestion_control=htcp # recommended for hosts with jumbo frames enabled net.ipv4.tcp_mtu_probing=1 # sysctl -p /tmp/tunetcp.conf
Increase Transmit Queue Length for 10G NICs with:
/sbin/ifconfig em1 txqueuelen 10000 /sbin/ifconfig em2 txqueuelen 10000
To make the change permanent, also append the above 2 lines to /etc/rc.d/rc.local.
Note ifconfig is obsolete. The replacement is ip:[2]
/sbin/ip link set em1 txqueuelen 10000 /sbin/ip link set em1 txqueuelen 10000
To make the change permanent, instead of modifying /etc/rc.d/rc.local, we can create a udev rules file /etc/udev/rules.d/75-custom-txqueuelen.rules:
# # custom txqueuelen parameter value # KERNEL=="em[1,2]", RUN+="/sbin/ip link set %k txqueuelen 10000"
To apply the change on the fly, run:
/sbin/udevadm trigger