Qubes - hpaluch/hpaluch.github.io GitHub Wiki

Qubes OS

Finally I got working Qubes on both VirtualBox (quite easy thanks to official HowTo) and on my old MSI-7250 dual-core machine - took me several days, because only 2nd integrated LAN works properly under Xen PCI passthrough...

Qubes OS is interesting Linux solution based around Xen and several VMs that defines Network (sys-net), firewall (sys-firewall) and several kinds of environments - for example some may have only local disk access (but not network access) and others.

See nice architecture image and description on: https://www.qubes-os.org/intro/

Even Dom0 has intentionally no Network access (but it has to pass-through required Network devices to sys-net VM) - it increases security but puts significant constraints on supported hardware (see text below).

Tested installation:

Strict hardware requirements

Xen uses (roughly) following hierarchy:

  • Xen Hypervisor - small layer booted directly from GRUB. You can see messages from this layer using xl dmesg command.
    • privileged VM called Dom0 (Dom is shortcut from Domain - antique name for "Virtual Machine") - it is operating system that "apparently" boots, however it is actually also VM. Normally only this Dom0 has access to hardware (Disks, Network, USB, ...) - but generally not in Qubes (continue reading)
    • also regular VMs are managed by Hypervisor, but management is done from Dom0 calling Hypervisor.
  • it means that for example top comand in Dom0 does not see CPU or memory usage from other VMs. One has to use xl top

Qubes has very strict hardware requirements - significantly more constraining than plain Xen. Why?

Because normally some hardware is passed-through to dedicated VM (instead of standard way - Dom0):

  • Network hardware is accessed from sys-net VM
  • USB hardware (mouse, keyboard) is similarly handled by sys-usb VM (optional)

It means that these devices must be passed through from Dom0 to specific VM. And it is exactly point where "fun" starts. Originally (version 3.x) only PV guests were supported, but they seems to be deprecated (in version 4.2 or later) - one reason is that PV guest can evade restriction by setting up rogue DMA transfers.

Official recommended way is via HVM guests but it requires 2 things:

However there is till very good VirtualBox HowTo which also applies to old bare metal boxes:

Install common notes

Both my targets (VirtualBox and old Dual Opteron machine) supports PV mode only:

  • HV mode requires NPT HAP/Shadow pages
  • HV passthrough requires IOMMU

Here is script that

  • sets PV mode to all VMs (excluding Dom0 where it has no sense)
  • sets vCPUs count to 1
  • has to be run in Dom0 Terminal or console (where you login after boot)
#!/bin/bash
set -euo pipefail
for i in `qvm-ls --raw-list`
do
	[ "$i" != dom0 ] || continue
	set -x
	qvm-prefs --set $i virt_mode pv
	qvm-prefs --set $i vcpus 1
	set +x
done
exit 0

Install in VirtualBox

There is very good howto at:

My only note:

  • Remember to NOT use virtio-net for network - it will always fail with Reset errors (even with workarounds). Intel Network card works fine.

Install on unsupported bare metal K9N - Platinum (MSI-7250)

Important:

  • do NOT select sys-usb (limiting USB access to single Qube/VM called sys-usb). I was unable to setup working USB controller passthrough on this machine - just kernel panic when booting sys-usb

Setup is similar to VirtualBox (must use always PV with workarounds), but there is one big catch:

Only 2nd integrated LAN card (MCP 55 Ethernet) works properly in sys-net VM. First card is simply malfunctioning with "Invalid MAC address 00:00:00:00:00:00" error and then with interrupts problem (not receive interrupts until other Qubes are started - and network will be flipping anyway).

So I passthrough both integrated LAN cards, but use only 2nd one (disconnect network from 1st one).

2nd integrated LAN card works like charm - even NetowrkManager will quickly catch-up with DHCP assigned IP address.

Here is relevant output from lspci in Dom0:

00:08.0 Bridge: NVIDIA Corporation MCP55 Ethernet (rev a2)
00:09.0 Bridge: NVIDIA Corporation MCP55 Ethernet (rev a2)

Only LAN card at 00:09.0 works properly - has usual MAC address and interrupts work out of the box.

Here is complete output for sys-net qube from command qvm-prefs --get sys-net:

audiovm             D  dom0
auto_cleanup        D  False
autostart           -  True
backup_timestamp    U
debug               D  False
default_dispvm      D  default-dvm
default_user        D  user
dispid              -  6523
dns                 D  10.139.1.1 10.139.1.2
gateway             D  10.138.25.173
gateway6            D
guivm               D  dom0
icon                D  servicevm-red
include_in_backups  D  True
installed_by_rpm    D  False
ip                  D  10.138.25.173
ip6                 D
kernel              D  6.6.36-1.fc37
kernelopts          -   iommu=soft swiotlb=2048 pci=nomsi
keyboard_layout     D  us++
klass               D  DispVM
label               -  red
mac                 D  00:16:3e:5e:6c:00
management_dispvm   D  default-mgmt-dvm
maxmem              -  0
memory              -  400
name                -  sys-net
netvm               -  None
provides_network    -  True
qid                 -  5
qrexec_timeout      D  60
shutdown_timeout    D  60
start_time          D  1725294380.7
stubdom_mem         U
stubdom_xid         D  -1
template            -  default-dvm
updateable          D  False
uuid                -  c78ad882-bad3-4914-ae7c-c799c36691ef
vcpus               -  1
virt_mode           -  pv
visible_gateway     D
visible_gateway6    D
visible_ip          D  10.138.25.173
visible_ip6         D
visible_netmask     D
xid                 D  3

Most important are kernelopts and virt_mode

And here is HCL command output:

Qubes release 4.2.2

Brand:		MSI
Model:		MS-7250
BIOS:		V1.11

Xen:		4.17.4
Kernel:		6.6.36-1

RAM:		8191 Mb

CPU:
  AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
Chipset:

VGA:
  NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2) (prog-if 00 [VGA controller])

Net:
  NVIDIA Corporation MCP55 Ethernet [10de:0373] (rev a2)
  NVIDIA Corporation MCP55 Ethernet [10de:0373] (rev a2)

SCSI:
  KINGSTON SA400S3 Rev: B1D2
  DVDRAM GSA-H12N  Rev: UL01

HVM:		Active
I/O MMU:	Not active
HAP/SLAT:	No
TPM:		Device not found
Remapping:	no
Certified:	no

Qubes HCL Files are copied to: 'dom0'
	Qubes-HCL-MSI-MS_7250-20240902-184429.yml		- HCL Info

Post Install: Setting network

Privileged Xen Dom0 has NO network access by design (see https://groups.google.com/g/qubes-users/c/c2RyhLmTCm4) Correct way is described here:

You have to understand these basics:

  • Dom0 has no network access for security reasons. However it has USB device access (if you did not create sys-usb on setup stage - which is only option in my case)
  • sys-net Qube is only VM that has direct access to Networking hardware and is running NetworkManager
  • sys-firewall is connected to sys-net (via preference netvm)
  • all Qubes with enabled network should connect via sys-firewall to network (their netvm preference is set to sys-firewall
  • for example I use Qube personal - it has unlimited Internet access by Default and there is even Firefox.

Keeping system up-to-date

Open Qube Manager GUI

  • select Dom0 Qube and click on Update
  • select your VM template (fedora-40-xfce) in my case - start it up, wait for some time and then again Select it and click on Update - if you just click on Update it may behave erratically until VM is fully booted
  • restart whole system to ensure that both Dom0 and VMs are updated

Installing additional packages

Boot your template (in my case fedora-40-xfce) run Terminal there (in main Menu go to Templates - fedora-40-xfce -> Xfce Terminal and install packages using plain dnf command, for example:

# net-tools provides "netstat", "bind-utils" provides "nslookup"
sudo dnf install mc net-tools bind-utils

After upgrade just reboot all Qubes that are attached to that template (for example personal Qube)

Data exchange

Is possible even from GUI using Global Clipboard and File Manager, please see:

Getting sources

Getting Cubes Dom0 kernel sources:

  • in my case uname -r returns: 5.15.94-1.qubes.fc32.x86_64
  • trying:
mkdir -p ~/git-src
cd ~/git-src/
git clone https://github.com/QubesOS/qubes-linux-kernel.git
cd qubes-linux-kernel/
git tag -l | fgrep 5.15.94
# returns: v5.15.94-1
git checkout v5.15.94-1
# Hmm, but don't know where to find all variables...