FreeBSD Network Tuning - shawfdong/hyades GitHub Wiki
The default installation of FreeBSD is pretty good and will work well for typical system. However, the default settings appear to be selected as to achieve good performance on Gigabit or slower network hardware. To achieve optimal performance on 10GE or faster network cards, one must tune, tweak and optimize FreeBSD's network stack.
There are a few excellent guides on tuning FreeBSD on the internet[1][2][3][4]. In this article, we document our tweaks of the 2 FreeBSD systems, Ambrosia & Coronis. Each system is equipped with a dual-port Intel X520-DA2 10 Gigabit Network Adapter and a dual-port Intel I350 Gigabit Ethernet Controller.
Here we collect and tabulate the default values of selected kernel parameters that are relevant to network performance. One can immediately see that many default values have been significantly increased in FreeBSD 10.x.
| parameter | FreeBSD 9.2 | FreeBSD 10.1 |
|---|---|---|
| kern.maxvnodes | 1095802 | 3339551 |
| kern.maxproc | 6164 | 201940 |
| kern.maxfiles | 12328 | 6289573 |
| kern.maxusers | 384 | 12620 |
| kern.ipc.nmbclusters | 25600 | 12255534 |
| kern.ipc.nmbjumbop | 12800 | 6127766 |
| kern.ipc.nmbjumbo9 | 6400 | 5446902 |
| kern.ipc.nmbjumbo16 | 3200 | 4085176 |
| kern.ipc.maxsockets | 25600 | 6289575 |
| kern.ipc.maxsockbuf | 2097152 | 2097152 |
| kern.ipc.somaxconn | 128 | 2048 |
| net.inet.tcp.sendbuf_auto | 1 | 1 |
| net.inet.tcp.recvbuf_auto | 1 | 1 |
| net.inet.tcp.sendspace | 32768 | 32768 |
| net.inet.tcp.recvspace | 65536 | 65536 |
| net.inet.tcp.sendbuf_max | 2097152 | 2097152 |
| net.inet.tcp.recvbuf_max | 2097152 | 2097152 |
| net.inet.tcp.sendbuf_inc | 8192 | 8192 |
| net.inet.tcp.recvbuf_inc | 16384 | 16384 |
| net.inet.tcp.tcbhashsize | 32768 | 2097152 |
| net.inet.ip.intr_queue_maxlen | 256 | 256 |
| net.route.netisr_maxqlen | 256 | 256 |
| hw.igb.max_interrupt_rate | 8000 | 8000 |
| hw.ix.max_interrupt_rate | 31250 | 31250 |
| hw.igb.rxd | 1024 | 1024 |
| hw.igb.txd | 1024 | 1024 |
| hw.ix.txd | 2048 | 2048 |
| hw.ix.rxd | 2048 | 2048 |
| hw.igb.num_queues | 0 | 0 |
| hw.ix.num_queues | 8 | 8 |
The file loader.conf contains descriptive information on bootstrapping the system. Through it one can specify the kernel to be booted, parameters to be passed to it, and additional modules to be loaded. The defaults are specified in /boot/defaults/loader.conf.
/boot/loader.conf for Ambrosia (running FreeBSD 9.3):
kern.ipc.nmbclusters=262144 kern.ipc.nmbjumbop=262144 kern.ipc.nmbjumbo9=65536 kern.ipc.nmbjumbo16=32768 kern.maxusers=1024 # number of hash table buckets to handle incoming TCP connections net.inet.tcp.tcbhashsize=65536 # H-TCP optimized congestion control algorithm for high speed networks with high latency cc_htcp_load="YES" zfs_load="YES"
/boot/loader.conf for Coronis (running FreeBSD 10.1):
# H-TCP optimized congestion control algorithm for high speed networks with high latency cc_htcp_load="YES" zfs_load="YES"
Note:
- The default values for parameters controlling the memory buffers mbufs and mbuf clusters in FreeBSD 9.x are too low and must be adjusted
- One can run netstat -m to show statistics on memory buffers
- The default TCP congestion-avoidance algorithm in FreeBSD is TCP New Reno.
- CUBIC and HTCP are better congestion control algorithms. We'll use HTCP, so we load the kernel module cc_htcp.ko on system startup.
The file sysctl.conf is read in when the system goes into multi-user mode to set default settings for the kernel. It is the primary tuning & optimization file for FreeBSD.
/etc/sysctl.conf for Ambrosia (running FreeBSD 9.3):
# $FreeBSD: releng/9.3/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $ # kern.maxfiles=51200 kern.ipc.somaxconn=2048 # set to at least 16MB for 10GE hosts kern.ipc.maxsockbuf=16777216 # socket buffers net.inet.tcp.recvspace=4194304 net.inet.tcp.sendspace=2097152 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendbuf_auto=1 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_inc=524288 # security security.bsd.see_other_uids=0 security.bsd.see_other_gids=0 # drop UDP packets destined for closed sockets net.inet.udp.blackhole=1 # drop TCP packets destined for closed sockets net.inet.tcp.blackhole=2 # ipfw net.inet.ip.fw.verbose_limit=3 # H-TCP congestion control algorithm net.inet.tcp.cc.algorithm=htcp # maximum incoming and outgoing IPv4 network queue sizes net.inet.ip.intr_queue_maxlen=2048 net.route.netisr_maxqlen=2048 # net.inet.tcp.tso=0 # net.bpf.optimize_writers=1
/etc/sysctl.conf for Coronis (running FreeBSD 10.1):
# $FreeBSD: releng/10.1/etc/sysctl.conf 112200 2003-03-13 18:43:50Z mux $ # # Ensure ZFS uses 4k block size vfs.zfs.min_auto_ashift=12 kern.ipc.somaxconn=2048 # set to at least 16MB for 10GE hosts kern.ipc.maxsockbuf=16777216 # socket buffers net.inet.tcp.recvspace=4194304 net.inet.tcp.sendspace=2097152 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.sendbuf_auto=1 net.inet.tcp.recvbuf_auto=1 net.inet.tcp.sendbuf_inc=16384 net.inet.tcp.recvbuf_inc=524288 # security security.bsd.see_other_uids=0 security.bsd.see_other_gids=0 # drop UDP packets destined for closed sockets net.inet.udp.blackhole=1 # drop TCP packets destined for closed sockets net.inet.tcp.blackhole=2 # ipfw net.inet.ip.fw.verbose_limit=3 # H-TCP congestion control algorithm net.inet.tcp.cc.algorithm=htcp # maximum incoming and outgoing IPv4 network queue sizes net.inet.ip.intr_queue_maxlen=2048 net.route.netisr_maxqlen=2048
Note the default values for parameters controlling socket buffers are too low for 10GE network cards, and must be adjusted.