Qubes - hpaluch/hpaluch.github.io GitHub Wiki

Qubes OS

Finally I got working Qubes on both VirtualBox (quite easy thanks to official HowTo) and on my old MSI-7250 dual-core machine - took me several days, because only 2nd integrated LAN works properly under Xen PCI passthrough...

Qubes OS is interesting Linux solution based around Xen and several VMs that defines Network (sys-net), firewall (sys-firewall) and several kinds of environments - for example some may have only local disk access (but not network access) and others.

See nice architecture image and description on: https://www.qubes-os.org/intro/

Even Dom0 has intentionally no Network access (but it has to pass-through required Network devices to sys-net VM) - it increases security but puts significant constraints on supported hardware (see text below).

Tested installation:

download page: https://www.qubes-os.org/downloads/
download url: https://mirrors.edge.kernel.org/qubes/iso/Qubes-R4.2.2-x86_64.iso
sha256: 939df3096cbcafa784b8d9866d9221075b66e69da031a3340bfd7da7c881b08b

Strict hardware requirements

Xen uses (roughly) following hierarchy:

Xen Hypervisor - small layer booted directly from GRUB. You can see messages from this layer using xl dmesg command.
- privileged VM called Dom0 (Dom is shortcut from Domain - antique name for "Virtual Machine") - it is operating system that "apparently" boots, however it is actually also VM. Normally only this Dom0 has access to hardware (Disks, Network, USB, ...) - but generally not in Qubes (continue reading)
- also regular VMs are managed by Hypervisor, but management is done from Dom0 calling Hypervisor.
it means that for example top comand in Dom0 does not see CPU or memory usage from other VMs. One has to use xl top

Qubes has very strict hardware requirements - significantly more constraining than plain Xen. Why?

Because normally some hardware is passed-through to dedicated VM (instead of standard way - Dom0):

Network hardware is accessed from sys-net VM
USB hardware (mouse, keyboard) is similarly handled by sys-usb VM (optional)

It means that these devices must be passed through from Dom0 to specific VM. And it is exactly point where "fun" starts. Originally (version 3.x) only PV guests were supported, but they seems to be deprecated (in version 4.2 or later) - one reason is that PV guest can evade restriction by setting up rogue DMA transfers.

Official recommended way is via HVM guests but it requires 2 things:

that IOMMU virtualization is supported by hardware (both CPU and sometime even BIOS support is required)
additionally latest Xen (not sure from what version) now requires that at least HAP or Shadow paging is required - see this commit that I'm unable to understand: https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=d974f321a7996e149e71f8773ac8d3448d3a07b7

However there is till very good VirtualBox HowTo which also applies to old bare metal boxes:

https://forum.qubes-os.org/t/how-to-install-and-use-qubes-os-inside-virtualbox/23453

Install common notes

Both my targets (VirtualBox and old Dual Opteron machine) supports PV mode only:

HV mode requires NPT HAP/Shadow pages
HV passthrough requires IOMMU

Here is script that

sets PV mode to all VMs (excluding Dom0 where it has no sense)
sets vCPUs count to 1
has to be run in Dom0 Terminal or console (where you login after boot)

#!/bin/bash
set -euo pipefail
for i in `qvm-ls --raw-list`
do
	[ "$i" != dom0 ] || continue
	set -x
	qvm-prefs --set $i virt_mode pv
	qvm-prefs --set $i vcpus 1
	set +x
done
exit 0

Install in VirtualBox

There is very good howto at:

https://forum.qubes-os.org/t/how-to-install-and-use-qubes-os-inside-virtualbox/23453

My only note:

Remember to NOT use virtio-net for network - it will always fail with Reset errors (even with workarounds). Intel Network card works fine.

Install under KVM

Follow above VirtualBox guide and additionally:

use SATA for both HDD and CD (virtio has issues)
use e1000 for older Qubes (R4.3.0-rc2) or rtl8139 for recent (at least R4.3.0-rc2).

Note: if you get famous error Unable to reset PCI device 0000:xx:xx.x no FLR, PM reset or bus reset available. despite all measures described in VirtualBox guide or on https://forum.qubes-os.org/t/sys-net-not-starting-because-it-is-unable-to-reset-pci-device/15056 and replace e1000 card with rtl8139. I experienced that issue with recent Qubes-R4.3.0-rc2-x86_64.iso

But beware!

if you just remove old LAN card and/replace and start VM under KVM you will be in big troubles!
attempt to edit settings for sys-net will simply crash
also qvm-pci command in dom0 will not help
run Terminal in dom0 and:
in such case you have to manually edit /var/lib/qubes/qubes.xml and lookup and remove line with original Intel e1000 card - it will have ID 0x8086:0x100e - remove whole such line.
and reboot Qubes completely

Tip: once you have Qubes running under LibVirt/KVM you can use Virtiofs as shared folder between Host and Dom0 VM. It is handy when you want to copy and paste commands and it will work even without network (or any other cube running).

Install on unsupported bare metal K9N - Platinum (MSI-7250)

Important:

do NOT select sys-usb (limiting USB access to single Qube/VM called sys-usb). I was unable to setup working USB controller passthrough on this machine - just kernel panic when booting sys-usb

Setup is similar to VirtualBox (must use always PV with workarounds), but there is one big catch:

Only 2nd integrated LAN card (MCP 55 Ethernet) works properly in sys-net VM. First card is simply malfunctioning with "Invalid MAC address 00:00:00:00:00:00" error and then with interrupts problem (not receive interrupts until other Qubes are started - and network will be flipping anyway).

So I passthrough both integrated LAN cards, but use only 2nd one (disconnect network from 1st one).

2nd integrated LAN card works like charm - even NetowrkManager will quickly catch-up with DHCP assigned IP address.

Here is relevant output from lspci in Dom0:

00:08.0 Bridge: NVIDIA Corporation MCP55 Ethernet (rev a2)
00:09.0 Bridge: NVIDIA Corporation MCP55 Ethernet (rev a2)

Only LAN card at 00:09.0 works properly - has usual MAC address and interrupts work out of the box.

Here is complete output for sys-net qube from command qvm-prefs --get sys-net:

audiovm             D  dom0
auto_cleanup        D  False
autostart           -  True
backup_timestamp    U
debug               D  False
default_dispvm      D  default-dvm
default_user        D  user
dispid              -  6523
dns                 D  10.139.1.1 10.139.1.2
gateway             D  10.138.25.173
gateway6            D
guivm               D  dom0
icon                D  servicevm-red
include_in_backups  D  True
installed_by_rpm    D  False
ip                  D  10.138.25.173
ip6                 D
kernel              D  6.6.36-1.fc37
kernelopts          -   iommu=soft swiotlb=2048 pci=nomsi
keyboard_layout     D  us++
klass               D  DispVM
label               -  red
mac                 D  00:16:3e:5e:6c:00
management_dispvm   D  default-mgmt-dvm
maxmem              -  0
memory              -  400
name                -  sys-net
netvm               -  None
provides_network    -  True
qid                 -  5
qrexec_timeout      D  60
shutdown_timeout    D  60
start_time          D  1725294380.7
stubdom_mem         U
stubdom_xid         D  -1
template            -  default-dvm
updateable          D  False
uuid                -  c78ad882-bad3-4914-ae7c-c799c36691ef
vcpus               -  1
virt_mode           -  pv
visible_gateway     D
visible_gateway6    D
visible_ip          D  10.138.25.173
visible_ip6         D
visible_netmask     D
xid                 D  3

Most important are kernelopts and virt_mode

And here is HCL command output:

Qubes release 4.2.2

Brand:		MSI
Model:		MS-7250
BIOS:		V1.11

Xen:		4.17.4
Kernel:		6.6.36-1

RAM:		8191 Mb

CPU:
  AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
Chipset:

VGA:
  NVIDIA Corporation GT218 [GeForce 210] [10de:0a65] (rev a2) (prog-if 00 [VGA controller])

Net:
  NVIDIA Corporation MCP55 Ethernet [10de:0373] (rev a2)
  NVIDIA Corporation MCP55 Ethernet [10de:0373] (rev a2)

SCSI:
  KINGSTON SA400S3 Rev: B1D2
  DVDRAM GSA-H12N  Rev: UL01

HVM:		Active
I/O MMU:	Not active
HAP/SLAT:	No
TPM:		Device not found
Remapping:	no
Certified:	no

Qubes HCL Files are copied to: 'dom0'
	Qubes-HCL-MSI-MS_7250-20240902-184429.yml		- HCL Info

Post Install: Setting network

Privileged Xen Dom0 has NO network access by design (see https://groups.google.com/g/qubes-users/c/c2RyhLmTCm4) Correct way is described here:

https://forum.qubes-os.org/t/how-to-set-up-network/13473

You have to understand these basics:

Dom0 has no network access for security reasons. However it has USB device access (if you did not create sys-usb on setup stage - which is only option in my case)
sys-net Qube is only VM that has direct access to Networking hardware and is running NetworkManager
sys-firewall is connected to sys-net (via preference netvm)
all Qubes with enabled network should connect via sys-firewall to network (their netvm preference is set to sys-firewall
for example I use Qube personal - it has unlimited Internet access by Default and there is even Firefox.

Keeping system up-to-date

Open Qube Manager GUI

select Dom0 Qube and click on Update
select your VM template (fedora-40-xfce) in my case - start it up, wait for some time and then again Select it and click on Update - if you just click on Update it may behave erratically until VM is fully booted
restart whole system to ensure that both Dom0 and VMs are updated

Installing additional packages

Boot your template (in my case fedora-40-xfce) run Terminal there (in main Menu go to Templates - fedora-40-xfce -> Xfce Terminal and install packages using plain dnf command, for example:

# net-tools provides "netstat", "bind-utils" provides "nslookup"
sudo dnf install mc net-tools bind-utils

After upgrade just reboot all Qubes that are attached to that template (for example personal Qube)

Data exchange

Is possible even from GUI using Global Clipboard and File Manager, please see:

Getting sources

Getting Qubes Dom0 kernel sources:

in my case uname -r returns: 5.15.94-1.qubes.fc32.x86_64
trying:

mkdir -p ~/git-src
cd ~/git-src/
git clone https://github.com/QubesOS/qubes-linux-kernel.git
cd qubes-linux-kernel/
git tag -l | fgrep 5.15.94
# returns: v5.15.94-1
git checkout v5.15.94-1
# Hmm, but don't know where to find all variables...

Controversial things

In Qubes 4.2.4 there is GeoClue service and even agent in all Fedora 41 based VMs(!), because it is included in fedora-41-xfce Template. GeoClue is wired deeply in D-Bus and provides precise location to applications. Its creepy agent even autostarts on any session (and can be seen even on sys-net and/or sys-firewall (!). See discussion on:

https://forum.qubes-os.org/t/geoclue-demo-agent/13719/10

Privileged Dom0 uses extremely old Fedora 37 packages (as of Oct 2025 current stable Fedora is 42, and Beta is 43). This logically ask question how security updates are maintained in Dom0. When I installed latest updates to Dom0 the kernel build time was Feb 2025 - I suspect that there will be various security issues in such old kernel version).

Win XP Support

It is quite easy to install Win XP under Qubes (create HVM qube and boot from ISO), but it is quite hard to integrate XP with Qubes (not officially supported).

However at least USB/PCI passthrough is possible. Following steps are needed:

Download and install USB 3.0 NEC xHCI drivers in your XP guest. I recommend ones from Dell Website (should be trustworthy) link: https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=36x7d&lwp=rt (I used regular Qube with Firefox for Download and then put it on SMB NAS where XP could read it).
Next you have to enable QRexec service in "stub-domain" (this domain runs QEMU that emulates PC hardware for HVM guest). You have to run these commands in Domain-0:
```
dom0$ qvm-features VM_NAME stubdom-qrexec 1
dom0$ qvm-features VM_NAME # verify that stubdom-exec is really set to 1
```

Finally you have to use GUI or CLI in Domain-0 to assign USB device (Blue ray USB 3.0 drive from Verbatim) to VM:

dom0$ qvm-usb list

BACKEND:DEVID  DESCRIPTION                                        USED BY
sys-usb:3-1    USB_Storage: Verbatim 000000000264

dom0$ qvm-usb attach VM_NAME sys-usb:3-1

Building Qubes (VMs)

It is tricky process because from 4.3RC there is new builder (called V2) and it is even more poorly documented than previous builder.

Main project is on: https://github.com/QubesOS/qubes-builderv2

Requirements:

having installed and updated Qubes R4.3: Qubes-R4.3.0-x86_64.iso

First prepare custom Template VM (we will use it for both Disposable and development):

clone template with (new template has suffix qb=Qube Builder):
```
# run in dom0:
qvm-clone fedora-42-xfce fedora-42-xfce-qb
```
boot that new template fedora-42-xfce-qb create or copy script setup1.sh with contents:

#!/bin/bash
set -xeuo pipefail
# run in template: fedora-42-xfce-qb
cd
http_proxy=http://127.0.0.1:8082/ https_proxy=http://127.0.0.1:8082/ \
	curl -fLO https://raw.githubusercontent.com/QubesOS/qubes-builderv2/refs/heads/main/dependencies-fedora.txt
http_proxy=http://127.0.0.1:8082/ https_proxy=http://127.0.0.1:8082/ \
	curl -fLO https://raw.githubusercontent.com/QubesOS/qubes-builderv2/refs/heads/main/dependencies-fedora-qubes-executor.txt
# install Builder/Docker deps
sudo dnf install $(cat dependencies-fedora.txt)
# install Qubes Executor deps
sudo dnf install $(cat dependencies-fedora-qubes-executor.txt)
exit 0

And run it - confirm that you want install packages (2 times) - it should finish without errors.

Now shutdown fedora-42-xfce-qb (so we "commit" all changes for future AppVMs that we will create). Inside VM you can simply run sudo poweroff Remember to poll with qvm-ls in dom0 and wait until State of VM is Halted

Now we will create one AppVM (normal Qube for development) and one disposable template for Qubes Executor (it is paradoxically also AppVM but with specific property set to True):

# run in: dom0 as default user
qvm-create -C AppVM -t fedora-42-xfce-qb --label red work-qubesos
qvm-create -C AppVM -t fedora-42-xfce-qb --label red qubes-builder-dvm
qvm-prefs qubes-builder-dvm template_for_dispvms True

Note: you may need to restart Qube Manager to see proper (disposable) icon for Qube qubes-builder-dvm

Now start qube work-qubesos and run these commands inside (we will need one configuration file later):

# run in: work-qubesos as "user"
cd
git clone https://github.com/QubesOS/qubes-release-configs.git

# WARNING! Git repo below must be exactly under /home/user!
# That path is referenced from qubes-release-configs repo
git clone https://github.com/QubesOS/qubes-builderv2.git
cd qubes-builderv2
git submodule update --init

Now we will follow https://github.com/QubesOS/qubes-builderv2 Copy policy file from work-qubesos to dom0:

# run in: dom0 as default user
qvm-run -p work-qubesos cat qubes-builderv2/rpc/policy/50-qubesbuilder.policy |
   sudo tee /etc/qubes/policy.d/50-qubesbuilder.policy

TODO: Configure DVM from https://github.com/QubesOS/qubes-builderv2

Run qubes-builder-dvm and follow above GitHub guide:

# run in: qubes-builder-dvm
sudo mkdir -p /rw/bind-dirs/builder /rw/config/qubes-bind-dirs.d
echo "binds+=('/builder')" | sudo tee /rw/config/qubes-bind-dirs.d/builder.conf
echo 'mount /builder -o dev,suid,remount' | sudo tee -a /rw/config/rc.local
sudo poweroff

Finally run in dom0 (again from GitHub guide):

# run in: dom0
qvm-prefs work-qubesos default_dispvm qubes-builder-dvm

Now resize private volume size to 30GB at least for:

qubes-builder-dvm
work-qubesos

Using:

# run in: dom0
qvm-volume extend qubes-builder-dvm:private 30G
qvm-volume extend work-qubesos:private 30G

Now moment of truth: run work-qubesos and try these commands

# run in: work-qubesos
cd ~/qubes-builderv2
./qb --builder-conf example-configs/qubes-os-r4.3.yml -c builder-rpm -c builder-debian -c qubes-release package fetch

cd ~/qubes-release-configs/R4.3/
~/qubes-builderv2/qb --builder-conf qubes-os-r4.3-templates-itl.yml -t fedora-42-xfce template fetch prep build

On success you should find root.img as shown below:

[user@work-qubesos R4.3]$ ls -olhs ~/qubes-builderv2/artifacts/templates/qubeized_images/fedora-42-xfce/root.img

5.4G -rw-r--r--. 1 user 20G Mar 26 18:54 /home/user/qubes-builderv2/artifacts/templates/qubeized_images/fedora-42-xfce/root.img

[user@work-qubesos R4.3]$ file ~/qubes-builderv2/artifacts/templates/qubeized_images/fedora-42-xfce/root.img

/home/user/qubes-builderv2/artifacts/templates/qubeized_images/fedora-42-xfce/root.img: DOS/MBR boot sector, extended partition table (last)

[user@work-qubesos R4.3]$ fdisk -l ~/qubes-builderv2/artifacts/templates/qubeized_images/fedora-42-xfce/root.img

Disk /home/user/qubes-builderv2/artifacts/templates/qubeized_images/fedora-42-xfce/root.img: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F4796A2A-E377-45BD-B539-D6D49E569055

Device                                                                                   Start      End  Sectors  Size Type
/home/user/qubes-builderv2/artifacts/templates/qubeized_images/fedora-42-xfce/root.img1   2048   411647   409600  200M EFI Syst
/home/user/qubes-builderv2/artifacts/templates/qubeized_images/fedora-42-xfce/root.img2 411648   415743     4096    2M BIOS boo
/home/user/qubes-builderv2/artifacts/templates/qubeized_images/fedora-42-xfce/root.img3 415744 41940991 41525248 19.8G Linux fi

How to import and run template:

WARNING! You need volume with at least 20GB FREE space that has to be accessible from dom0. I have /mnt/host which is VirtioFS folder to KVM Host
run these commands in dom0:

# run in: dom0
qvm-create -C TemplateVM -r /mnt/host/hp/root.img --label red  hp-f42-template
qvm-prefs hp-f42-template netvm sys-firewall

Now simply start hp-f42-template in Qube Manager click on Open Console, login as user (no password) and try to install something, for example:

# run in: hp-f42-template
sudo dnf install mc

It should work. Power-off TemplateVM with sudo poweroff in same Console.

Posted on: https://forum.qubes-os.org/t/building-f42-template-in-r4-3/40176

I used similar way to build archlinux template:

cd ~/qubes-release-configs/R4.3/
~/qubes-builderv2/qb --builder-conf qubes-os-r4.3-templates-community.yml -t archlinux template fetch prep build
# creates root fs:
# => ~/qubes-builderv2/artifacts/templates/qubeized_images/archlinux/root.img

You can use it same way as for F42 template to copy it to dom0 and create Qube Template from it

WARNING! Both Fedora and Arch is missing important package called qube-desktop-linux-common that is required to integrate application menu from Qube to Host

Here is fix for Arch template:

run your Arch template (in my case hp-arch-template Template qube).

inside run:

sudo pacman -S --needed qubes-vm-gui-common

restart Template
now in Qubes manager:
- select this template
- click on App Shortcuts
- click on Refresh applications
- you should see how Menu was populated with many applications
power off template - it is now ready for use

Building Gentoo Qube with forked repository

Building Gentoo Qube:

problem - related repository qubes-builder-gentoo is old and broken (expired GPG key)
we need to accept PR from saces that fixes it - from: https://github.com/QubesOS/qubes-builder-gentoo/pull/13 his repo: https://github.com/saces/qubes-builder-gentoo
you need to apply following patch to ~/qubes-release-configs to use forked repo:

diff --git a/R4.3/qubes-os-r4.3-templates-community.yml b/R4.3/qubes-os-r4.3-templates-community.yml
index 30c8d8c..b899004 100644
--- a/R4.3/qubes-os-r4.3-templates-community.yml
+++ b/R4.3/qubes-os-r4.3-templates-community.yml
@@ -51,7 +51,11 @@ components:
       packages: False
       fetch-versions-only: false
       branch: main
+      url: https://github.com/saces/qubes-builder-gentoo.git
+      verification-mode: less-secure-signed-commits-sufficient
       maintainers:
+        # saces
+        - 8D69A04F7F0602B183037FA22D1B511919F86901
         # fepitre's @qubes-os.org
         - 9FA64B92F95E706BF28E2CA6484010B5CDC576E2
         # fepitre's @invisiblethingslab.com

but that's not all, we must also import saces GPG public key using:

cd ~/qubesbuilder/plugins/fetch/keys/
curl -fLO https://github.com/saces.gpg
gpg saces.gpg  # extract public key ID
mv saces.gpg 8D69A04F7F0602B183037FA22D1B511919F86901.asc

finally you can try build:

cd ~/qubes-release-configs/R4.3/
~/qubes-builderv2/qb --builder-conf qubes-os-r4.3-templates-community.yml -t gentoo-xfce template fetch prep build