Isolating PCI devices - Qrivi/KVM GitHub Wiki

We need to prevent our host Linux from loading/using the specific PCI devices we want to passthrough to the guest VMs, so we will have their drivers replaced by a VFIO stub driver. This is how we achieve near-native performance on the guests.

For our convenience, open up a second terminal window in which we'll list our IOMMU groups and the PCI devices mapped to these groups. To do so, run the provided iommu.sh from the scripts folder.

sh ~/KVM/scripts/iommu.sh

If this script does not list any groups or PCI devices, IOMMU is not enabled or the hardware does not support it. The output should look as follows:

IOMMU Group 0:
	00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v6/7th Gen Core Processor Host Bridge/DRAM Registers [8086:591f] (rev 05)
IOMMU Group 1:
	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 05)
	01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef)
	01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
IOMMU Group 10:
	00:1b.4 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #21 [8086:a2eb] (rev f0)

The stub driver we'll be using is vfio-pci which is included in the Linux kernel. vfio-pci targets PCI devices by their vendor-device ID. These IDs are the numbers in between the square brackets after each device's name in the iommu.sh output.

As an IOMMU group is the smallest set of physical devices that can be passed to a virtual machine, you need to passthrough either none or either all the devices in a certain group. This implies that if a device you want to passthrough to a guest is in the same group as a device you need on the host, you're SOL (there are things you can try, but those are not in scope of this guide).

There is 1 exception! Depending on the CPU, processor-based PCI slots might not properly support isolation, in which case the slot itself will appear to be grouped with the devices occupying it (listed as "bridges" in the example output above). This is fine but these root ports should stay attached to the host to function properly. If not your mileage may vary.

If you have multiple identical hardware parts in your build (e.g. the same graphic card twice), they will share the same vendor-device ID which makes this ID unqualified to target a specific part. There's a workaround, but that's also not in scope of this guide.

Add a modprobe configuration file for vfio, in which we list the device IDs of the hardware we want to make available. You will need sudo to create and edit this file. Again, it is mandatory to list all the devices that are in the same group except the PCI root port.

sudo nano /etc/modprobe.d/vfio.conf

In the example below I am passing through the RX 570, both its HDMI display and HDMI audio module, but I'm not including the PCIe controller.

options vfio-pci ids=1002:67df,1002:aaf0s

After a reboot, the host should no longer pick up the hardware listed in the vfio.conf. So if e.g. you added a graphic card, it won't output any image anymore. To verify which devices are using the VFIO stub driver, you can execute:

lspci -nnk | grep -A1 -B2 -i vfio

The output should be similar to:

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef)
	Subsystem: ASUSTeK Computer Inc. Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1043:051d]
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
	Subsystem: ASUSTeK Computer Inc. Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1043:aaf0]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

All done! These devices are now ready for hardware passthrough!