SPDK NVMe - pykello/pykello.github.com GitHub Wiki

  • Setting up PCI devices for VFIO pass-through

  • Understanding VFIO and UIO Driver Framework

  • Kernel VFIO

  • Memory in DPDK

    • PA mode
      • Virtual memory layout matches physical memory layout
      • Good: does not require IOMMU
      • Bad: Requires root
    • VA mode
      • Physical memory layout is not followed
      • Uses IOMMU to remap physical memory
      • Does not require privileges
    • Drivers
      • UIO only supports PA mode
      • VFIO can use IOMMU

  • Running SPDK as non-privileged user

    • nvme list -v to list addresses
    • lspci -n to list vendor:device numbers
    • readlink "/sys/bus/pci/devices/0000:00:04.0/iommu_group" to check if IOMMU group assigned
    • After a VFIO IOMMU group is assigned, we can spdk access using chown spdk /dev/vfio/5
    • To enable iommu, add intel_iommu=on or amd_iommu=on to kernel parameters.
      • Verify by cat /proc/cmdline or dmesg | grep -e DMAR -e IOMMU or ls /sys/kernel/iommu_groups/
    • To bind a PCIe device to vfio driver: PCI_ALLOWED="0000:01:00.0" DRIVER_OVERRIDE=vfio-pci scripts/setup.sh