• DPDK setup
• Ubuntu setup
  • BIOS
  • PVN setup
    • Experiments
      • tuco
      • saul
  • Example 0: working config from Nefeli
    • acpi=off is the one to blame so let's get rid of it
  • Example 1: conservative config from Nefeli
  • Example 2: GENI
  • Example 3: GENI 2
  • Example 4: aggressive config
• DPDK in general
  • NIC
    • OVS DPDK
    • Linux drivers/kernel modules
  • huge page size
    • other hugepage example setup
  • DPDK performance tuning and Tuned
    • NUMA
    • vimdiff common_linuxapp-17.08 common_linuxapp-17.08.container
    • DPDK 2.2.x
    • DPDK 17.08
    • HT
  • 1. Build NetBricks
  • 2. DPDK setup
    • WIP--Node 1 (marshall):
    • WIP--Node 2 (mathers):
  • Performance tuning
  • NetBricks server hardware setup
• Disk IO
  • Disable disk/page cache so that it will do a disk write directly
    • cmd
    • links

https://stackoverflow.com/questions/13583146/whole-one-core-dedicated-to-single-process

DPDK setup

http://core.dpdk.org/download/

http://git.dpdk.org/dpdk/refs/tags

http://static.dpdk.org/rel/

https://git.dpdk.org/dpdk-stable/refs

Ubuntu setup

BIOS

ovs dpdk recommend

PVN setup

/etc/default/grub

Experiments

tuco

/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=10 nosoftlockup processor.max_cstate=0 intel_idle.max_cstate=0 intel_pstate=disable pcie_aspm.policy=performance noht transparent_hugepages=never isolcpus=2-6 irqaffinity=1 nohz_full=2-6 rcu_nocbs=2-6 apm=off"

saul

/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=40 nosoftlockup processor.max_cstate=0 intel_idle.max_cstate=0 intel_pstate=disable pcie_aspm.policy=performance noht transparent_hugepages=never isolcpus=2-6 irqaffinity=1 nohz_full=2-6 rcu_nocbs=2-6 apm=off"

NOTE(jethrosun): Allocating hugepage size 20 on a 64G mem machine might cause extra problem.

Example 0: working config from Nefeli

acpi=off is the one to blame so let's get rid of it

irqaffinity: 0-3
isolated cpu: 4-11

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=10 nosoftlockup processor.max_cstate=0 intel_idle.max_cstate=0 intel_pstate=disable pcie_aspm.policy=performance noht transparent_hugepages=never isolcpus=4-11 irqaffinity=0-3 nohz_full=4-11 rcu_nocbs=4-11 apm=off"

Example 1: conservative config from Nefeli

irqaffinity: 0-3
isolated cpu: 4-11

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=10 nosoftlockup processor.max_cstate=0 intel_idle.max_cstate=0 intel_pstate=disable pcie_aspm.policy=performance noht transparent_hugepages=never isolcpus=4-11 irqaffinity=0-3 nohz_full=4-11 rcu_nocbs=4-11 acpi=off apm=off"

Example 2: GENI

GRUB_CMDLINE_LINUX_DEFAULT="rcu_nocbs=1-4 rcu_nocb_poll=1 isolcpus=1-4 irqaffinity=0,5-11 nohz_full=1-4 highres=off tsc=perfect intel_iommu=on iommu=pt default_hugepagesz=2M hugepagesz=2M hugepages=4096 nmi_watchdog=0"

Example 3: GENI 2

GRUB_CMDLINE_LINUX_DEFAULT="acpi=off console=ttyS0 console=tty0 isolcpus=1-3 irqaffinity=0 rcu_nocbs=1-4 rcu_nocb_poll=1 clocksource=tsc tsc=reliable nohz_full=1-3 $HUGE selinux=0 enforcing=0 $DEBUG raid=noautodetect"

Example 4: aggressive config

isolated cpu: 2~10

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=10 nosoftlockup processor.max_cstate=0 intel_idle.max_cstate=0 intel_pstate=disable pcie_aspm.policy=performance noht transparent_hugepages=never isolcpus=2-10 irqaffinity=0-2 nohz_full=3-10 rcu_nocbs=3-10 acpi=off apm=off"

DPDK in general

doc
DPDK 17.08

NIC

I40E Poll Mode Driver Compiling and testing a PMD for a NIC

OVS DPDK

doc

Linux drivers/kernel modules

DPDK doc-linux driver

• uio_pci_generic
• igb_uio (I believe in NetBricks they use this kernel module)

$ apt install dpdk-igb-uio-dkms

• vfio-pci

huge page size

ubuntu setup guide

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=2048 iommu=pt intel_iommu=on isolcpus=1-27,29-55" reference

other hugepage example setup

Install DPDK (Ubuntu 16.04 and CentOS 7)

ClickController/prepare_host.sh

iCan-sharp-tools-docs/ovs_dpdk_install_steps

2017-05-22-nfv os config.md

org/network/ovs-install.org

lagopus/docs/how-to-allocate-1gb-hugepages.md

SO: Hugepagesize is not increasing to 1G in VM

Thread: Has anyone gotten 1G hugepages working in 18.04?

How to enable and config hugepage and transparent hugepages on RHEL,CentOS

DPDK performance tuning and Tuned

Chapter 8. NFV performance tuning with OVS-DPDK

Chapter 4. Configure DPDK Accelerated Open vSwitch (OVS) for Networking

Network Functions Virtualization Planning and Configuration Guide

Troubleshooting packet drop with NFV DPDK

Network Functions Virtualization Planning and Configuration Guide

NUMA

setup with 1 numa node?

vimdiff common_linuxapp-17.08 common_linuxapp-17.08.container

CONFIG_RTE_EAL_IGB_UIO=y
CONFIG_RTE_KNI_KMOD=y

DPDK 2.2.x

doc

DPDK 17.08

doc

HT

https://unix.stackexchange.com/questions/33450/checking-if-hyperthreading-is-enabled-or-not/33509#33509

1. Build NetBricks

This should be trivial since we can use our script and then is just building the crate.

2. DPDK setup

WIP--Node 1 (marshall):

WIP--Node 2 (mathers):

Performance tuning

not currently tested

1. scripts/tuning
2. RH tuned? link 1 link 2 link 3

NetBricks server hardware setup

• disable hyperthreading
• all cores are running at constant 2.6 GHz
• enable hardware virtualization like Intel VT

Cores:

• depend on how many cores are assigned for the container

Disk IO

Disable disk/page cache so that it will do a disk write directly

cmd

hdparm -W 0 /dev/sda

https://unix.stackexchange.com/questions/34449/how-to-deliberately-fragment-a-file

links

SO1 SO2 SO3