Dedicated GPU on VM - powwu/pso2-na-linux GitHub Wiki

Enabling IOMMU

To start, first ensure that VT-d/AMD-Vi are enabled in your BIOS.

Then, set a kernel parameter (by editing the GRUB_CMDLINE_LINUX_DEFAULT entry in /etc/default/grub):

For Intel CPUs (VT-d) set intel_iommu=on
For AMD CPUs (AMD-Vi) set `amd_iommu=on``

After a reboot, you can verify that it worked by running dmesg | grep -i -e DMAR -e IOMMU:

[    0.000000] ACPI: DMAR 0x00000000BDCB1CB0 0000B8 (v01 INTEL  BDW      00000001 INTL 00000001)
[    0.000000] Intel-IOMMU: enabled
[    0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.028950] IOAPIC id 8 under DRHD base  0xfed91000 IOMMU 1
[    0.536212] DMAR: No ATSR found
[    0.536229] IOMMU 0 0xfed90000: using Queued invalidation
[    0.536230] IOMMU 1 0xfed91000: using Queued invalidation
[    0.536231] IOMMU: Setting RMRR:
[    0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]
[    0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]
[    0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]
[    0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]
[    0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    2.182790] [drm] DMAR active, disabling use of stolen memory

Run the following script to determine your IOMMU groups:

#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

Look for the group with your GPU in it. If the group has more than just the GPU, you'll have to pass through all of the devices in the group in the step below. Otherwise, just pass through the GPU. Take note of the PCI IDs (example 10de:13c2) as you'll need them later.

Binding VFIO to your GPU

Edit your kernel parameters once more, and add the following:

vfio-pci.ids=[IDS]

For example:

vfio-pci.ids=10de:13c2

vfio-pci.ids=10de:13c2,10de:0fbb

Then, edit /etc/mkinitcpio.conf. Add the following to the MODULES entry:

vfio_pci vfio vfio_iommu_type1 vfio_virqfd

Additionally, ensure that modconf is present in the HOOKS entry. Then, regenerate the initramfs. If you are on a Debian-based distro (i.e Ubuntu), you may want to see this instead.

You can ensure it works by running:

dmesg | grep -i vfio

Example output:

[    0.329224] VFIO - User Level meta-driver version: 0.3
[    0.341372] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000
[    0.354704] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000
[    2.061326] vfio-pci 0000:06:00.0: enabling device (0100 -> 0103)

Then, edit your virtual machine. Add your PCI host devices (your GPU, and anything else you passed through to VFIO). Remove the video devices (Spice server first, then QXL). Make sure to pass through a USB keyboard and mouse as well! (Alternatively, you could use toggling via evdev.) Plug your GPU into a monitor, and start the VM. If all went according to plan, your VM should be outputting to a monitor.

Yet another fork in the road

Now from here, you have some options.

If your GPU is successfully connected to the VM, but not outputting to the monitor, then you can try setting up Looking Glass in the Integrated GPU on VM page. You can also follow this if you don't have a second monitor/second output to plug into.
If all is working well, then continue to the Post-complete install page for instructions on installing PSO2.