This section is here for performance tweaking to get the most out of your virtual machine, however this is also where things can go wrong or because most systems will not need to do this.

You are expected to familiar with how to work with grub and the root rescue shell in case anything goes wrong so you can repair your system in case you changed something you should not have changed.

You are also expected to know how a computer works and how the software works with the hardware.

In short, you need to know or understand what you are doing and you need to be capable of fixing your errors yourself if you do something wrong.

You have been warned.

Pinning the cpus or setting CPU affinity helps in many cases to give stable performance in the virtual machine. Usually I pin 2-6 cores to the virtual machine depending on how many host CPU cores I have, if I have 4 cores I pin 2 cores, if I have 6 cores I pin 4 cores, etc.

I also only give the virtual machine access to the same amount of cores that I pin.

Note: Pinning threads work the same way

1. Shut down the virtual machine and open the virtual machine config, as this cannot be done through the GUI

2. Locate the “<vcpu” tag, if you have chosen a simple topology (1 socket, x cores, 1 thread) it should be very simple to pin the CPU cores, the line should look something like this.

<vcpu placement='static'>2</vcpu>

3. What we want to do is add cpuset='0-1' after 'static', this will expose core 0 to 1 to the virtual machine. You can also expose separate cores by instead provide a comma separated list of cores.
   Like lets say I want to expose core 0 and 2 then I would write cpuset='0,2' instead.

   The line should look something like this now.

<vcpu placement='static' cpuset='0-1'>2</vcpu>

4. Now we need to fine tune our cores and pin them to virtual cores by assigning specific cores to specific virtual cores. This can be done by adding this below the <vcpu line.

<cputune>
  <vcpupin vcpu='0' cpuset='0'/>
  <vcpupin vcpu='1' cpuset='1'/>
  <emulatorpin cpuset='0-1'/>
</cputune>

5. This tells vcpu 0 to only use CPU core 0 and vcpu 1 to only use CPU core 1, while the emulatorpin tag will pin the CPU cores we provided in the vcpu tag to the QEmu process from my understanding as the documentation does not cover it much.

   For more information about vcpu and thread pinning check the full documentation.

In most cases you do not need to delve into this, but if you have a Threadripper system or any other CPU with multiple NUMA nodes, this will be of great help to you to get even more performance out of the virtual environment! Make sure your Memory Interleaving setting in the UEFI is set to NUMA mode instead of UMA mode, or set to “Channel” instead of “Die”.

1. First we need to install hwloc so we can use lstopo, this will be extremely useful to understand your CPU topology.

2. Next we launch lstopo

3. You will get a window similar to this

4. If you look carefully, the graphics card I stubbed earlier is on NUMA node 0 (remember I stubbed the PCI-e ids 10de:1b82 and 10de:10f0) so preferably I want to pin only cpus from NUMA node 0 to my virtual machine (see previous section) and only allocate memory from NUMA node 0 to the virtual machine, so let us do exactly that!

5. Shut down your virtual machine and open the virtual machine config.

6. Write down the amount of memory you have given the machine (this is written in KiB in the memory tag almost at the very top of the file) as you will need this soon.

7. Now locate the “</cputune>” end tag and paste the contents of the box below it (set nodeset to 1 if the graphic card you gave the virtual machine is on NUMA node 1)

   Note: If the graphic card you gave the virtual machine is located on NUMA node 1 then this is a good time to edit your CPU pins to only use CPU cores from NUMA node 1.

<numatune>
  <memory mode='strict' nodeset='0'/>
</numatune>

8. This will make the machine fail to start if it cannot allocate all the memory on the specified NUMA node, there are more mode policies you can use as specified by the documentation. However strict will give you the most consistent performance as you avoid having to cross communicate between the NUMA nodes.

9. Now locate the “<cpu” tag and add this inside it, edit the memory to match the amount of memory you have allocated to the virtual machine.

   The below configuration uses CPU core 0 to 1 and 8GB memory located on NUMA node 0.

<numa>
  <cell id='0' cpus='0-1' memory='8388608' unit='KiB'/>
</numa>

10. The final configuration should look something like this if you have used a manual topology in the CPU configuration in the Virtual Machine Manager.

<cpu mode='custom' match='exact' check='partial'>
  <model fallback='allow'>EPYC-IBPB</model>
  <topology sockets='1' cores='2' threads='1'/>
  <numa>
    <cell id='0' cpus='0-1' memory='8388608' unit='KiB'/>
  </numa>
</cpu>

11. Once you are done you save the file with CTRL+X.

Some anti-cheat systems will flag you (or outright ban you) if they detect you run the game inside a VM. Please check if this is needed for your game first and be aware a future update might make the anti-cheat system detect your VM again, in worst case ban you. My recommendation is to stay away from games like these in general. But if you feel like risking it, here is how to mask your hypervisor even more.

Locate your "<cpu" tag and change your <cpu mode="something" to <cpu mode="host-passthrough" or <cpu mode="host-model". Then paste this under the "<topology tag (if you do not have it, just make sure this is on a line before </cpu>).

<feature policy="disable" name="hypervisor"/>

For some CPUs you may also have to enable vpindex and synic in order to hide the hypervisor. Search for and add this on the line below it before

<vpindex state="on"/>
<synic state="on"/>

The Linux kernel will still be doing system interrupts on the isolated cores but the rest of the system will not touch them unless specifically told to do so, why would you do this? To avoid the host and guest from competing for the cores. However I do not recommend doing this on any system with less than 6 cores!

1. Edit the grub defaults file.

2. Add the isolcpus parameter to GRUB_CMDLINE_LINUX_DEFAULT, edit the parameter to contain the core numbers you do not want the system to touch.

   The example below isolates CPU cores 0,1 and 3, you may have more parameters listed than this example.

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=0,1,3"

3. Save the file with CTRL+X then run the update-grub command.

4. Reboot your system.

Use this instead of the above method to keep all cores active on the host system, but isolate them when you run your VM

1. Setup the qemu hooks following this guide
2. Make the paths needed (NOTE: replace win10 with your VM domain name)

3. Edit the file

4. Add the script (this allows the host to only use cores 6-11 and 18-23 on a 12 core 24 thread system) and save with CTRL+X followed by Y

#!/bin/bash                                                                                                                                                                                                                                                                              
HOST_CORES=6-11,18-23
systemctl set-property --runtime -- system.slice AllowedCPUs=${HOST_CORES}
systemctl set-property --runtime -- user.slice AllowedCPUs=${HOST_CORES}
systemctl set-property --runtime -- init.scope AllowedCPUs=${HOST_CORES}

5. Now do the same to revert the cpu core isolation when the VM releases its resources

6. Add the restore cpu cores script

#!/bin/bash                                                                                                                                                                                                                                                                              
HOST_CORES=0-23
systemctl set-property --runtime -- system.slice AllowedCPUs=${HOST_CORES}
systemctl set-property --runtime -- user.slice AllowedCPUs=${HOST_CORES}
systemctl set-property --runtime -- init.scope AllowedCPUs=${HOST_CORES}

7. Save with CTRL+X followed by Y and you are done.

NOTE: this will RESERVE the amount of pages in memory specifically for this VM, this means that if you create 10GB of Hugepages, your system will always use an extra 10GB of memory that will not be available for programs not configured/told to use Hugepages!

What is hugepages?
In short it lets you assign memory in 2MB (default) or 1GB chunks, so if you have something like a VM needing to access 4GB of memory at once, with 1GB hugepages you can make the CPU do only 4 lookups of 1GB to go through that 4GB memory instead of 2000 lookups of 2MB each which is what the default is set to with transparent hugepages.

Why use a script in rc.local for this? Easy to configure, also it is the quickest way I found to configure hugepages on consumer CPUs and Threadripper CPUs without issues

1. First install hugepages

1. Create a script called enable_hugepages in /usr/local/bin, save the file by pressing CTRL+X.

2. Paste in the following script (and edit the)

#!/bin/bash
# This is how many GB in huge pages you want to RESERVE (pre-allocate) for your virtual machine, keep in mind you cannot assign all available memory for your virtual machine, you will need at least 4GB available for your CPU/system overhead, adding too many hugepages will result in 0 hugepages being allocated
PAGES=10
sudo bash -c "echo $PAGES > /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages"

1. Create a script called enable_hugepages in /usr/local/bin, save the file by pressing CTRL+X.

2. Paste in the following script (and edit the)

# This is how many GB in huge pages you want to RESERVE (pre-allocate) for your virtual machine, keep in mind you cannot assign all available memory for your virtual machine,
# you will need at least 4GB available as normal memory for your NUMA node, this means on a NUMA node system with 32GB RAM, you can only use 12GB as hugepages for a single NUMA node, adding more will cause an error and you will end up with 0 hugepages
PAGES=16

# Assign the huge pages only to this NUMA node
# (same node that has the cpu cores used for your VM)
NUMA_NODE=0

# Assign the hugepages
sudo bash -c "echo $PAGES > /sys/devices/system/node/node$NUMA_NODE/hugepages/hugepages-1048576kB/nr_hugepages"

# Mount the hugepages
sudo mkdir /dev/hugepages1G
mount -t hugetlbfs -o pagesize=1G hugetblfs /dev/hugepages1G

1. Now add this as part of libvirtd's systemd startup

2. Add the following to the file and save with CTRL+X

[Service]
ExecStartPre=/usr/local/bin/enable_hugepages

1. Make a service file to fix permissions for hugepages in /dev

4. Paste the following and save with CTRL+X

[Unit]
Description=Systemd service to fix hugepages + qemu ram problems.
After=dev-hugepages.mount

[Service]
Type=simple
ExecStart=/usr/bin/chmod o+w /dev/hugepages/

[Install]
WantedBy=multi-user.target

5. Reload the systemd daemon and enable the fix.

6. Reboot your computer and hugepages will now be enabled on your system.

1. Edit your virtual machine config

1. Add this under the line starting with <currentMemory unit= and if you are using a Threadripper, EPYC or any other NUMA enabled cpu, then add nodeset="0" after unit="KiB" and replace 0 with the numanode where the hugepages are located.

<memoryBacking>
  <hugepages>
    <page size="1048576" unit="KiB" />
  </hugepages>
</memoryBacking>

This is a solution I have found to work on high core count systems. If you rely on super accurate clock timings then you should not change this and just accept the stutter until another solution is found!

1. Edit the grub defaults file.

2. Add clock=tsc to the GRUB_CMDLINE_LINUX_DEFAULT line. the line should look something like this and you may have more parameters than listed in this example.

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash clock=tsc"

3. Save the file with CTRL+X and then update grub.

The Level1Techs forum made me aware of this amazing tool to get easier access to mainline kernels for Ubuntu called Mainline, it lets you install and remove newer and older kernels at the click of a button, super handy if you use only hardware that is supported directly in the kernel!

If you are using an NVidia GPU as your host graphics and the proprietary driver, do not use Mainline as you will be at the mercy of NVidia about which kernels you can use, and those are usually the ones released through traditional Ubuntu updates.

But if you use an AMD GPU, using a newer kernel lets you get easy access to newer amdgpu drivers and other fixes for things you might have encountered, if any.

You can install Mainline with the commands below

When you open Mainline it should look like this once it has checked for any new kernels.

Kernels with the Ubuntu icon are kernels scheduled or already released for Ubuntu.