LibVirt is universal virtualization layer - it supports several hypervisors (QEMU+KVM, Xen, ...) and even containers (called LXC however it uses different toolset for these).

Note: because libvirt started as wrapper on Xen it uses (sometimes confusing) Xen terms - for example 'destroy' normally means "stop" or "shutdown" but Not delete... Similarly create means start (fortunately there is now also start command)

Installation under Fedora 40:

$ sudo dnf install libvirt-daemon-kvm libvirt-daemon-config-network libvirt-client-qemu

$ sudo dnf install virt-manager  # pouplar GUI for LibVirt

# ensure that creepyware geoclue2 is masked:
$ sudo systemctl mask --now geoclue
$ sudo rm -f /etc/xdg/autostart/geoclue-demo-agent.desktop

If you plan to use virt-manager remotely via SSH X11 Forwarding you have to install these packages on Server (with virt-manager):

# required
$ sudo dnf install xorg-x11-xauth

# optional - for testing
$ sudo dnf install xclock xterm

Also on your SSHd server enable X11 forwarding, for example, create /etc/ssh/sshd_config.d/99-local.conf with contents:

X11Forwarding yes

And restart SSHd with systemctl restart sshd

Also ensure that your SSH client requests X11 forwarding, here is excerpt from my ~/.ssh/config

Host f40lvm-s350
        HostName 192.168.0.X
        User USERNAME
        IdentityFile ~/.ssh/MY_SSH_KEY
        IdentitiesOnly yes
        HostKeyAlias f40lvm-s350
        ForwardX11 yes

Now try ssh connection from your client using alias, in my example: ssh f40lvm-s350 And try some X11 application, for example xclock (use Ctrl-C in terminal to quit)

GDK_BACKEND=x11 virt-manager

But before running virt-manager please read following chapter:

By default when you run any libvirt command it will use your private User connection (called "session"). So all networks and disks and VMs will be available to you only (and data stored under $HOME/.local)

To use always shared global (called system) connection you have to:

1. Add yourself to libvirt group using:

   sudo /usr/sbin/usermod -G libvirt -a $USER

2. Create file ~/.config/libvirt/libvirt.conf with following content:

   # see https://listman.redhat.com/archives/libvirt-users/2018-October/msg00067.html
   uri_default = "qemu:///system"

On Fedora I have to enable and start "libvirtd legacy service" with:

$ sudo systemctl enable --now libvirtd

To avoid virsh client error:

Failed to connect socket to '/var/run/libvirt/virtqemud-sock': No such file or directory

Remember that you should logout and login to ensure that you are member of group libvirt. Then you can for example try listing networks or pools:

$ virsh net-list

 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   yes         yes

$ virsh pool-list

 Name   State   Autostart
---------------------------

If there is no pool you can follow guide from Ubuntu MAAS KVM. Run as you (non-privileged user)

virsh pool-define-as default dir - - - - "/var/lib/libvirt/images"
virsh pool-autostart default
virsh pool-start default
virsh pool-list

   Name      State    Autostart
  -------------------------------
   default   active   yes

virsh pool-dumpxml default

   ...

At least when using NAT network we can pass option log-queries to dnsmasq that is used to provide both DHCP and DNS server for NAT network.

I followed various sources including: https://serverfault.com/a/1017645

Here is diff of default network - using virsh net-edit default or using virsh net-dumpxml default:

diff -u default.xml default-log-queries.xml
--- default.xml	2024-07-01 19:15:27.965334382 +0200
+++ default-log-queries.xml	2024-07-01 19:28:14.214143882 +0200
@@ -1,4 +1,4 @@
-<network connections='1'>
+<network xmlns:dnsmasq='http://libvirt.org/schemas/network/dnsmasq/1.0'>
   <name>default</name>
   <uuid>2c9b8477-9b7c-4ca5-8c20-f1fbcc7df3c3</uuid>
   <forward mode='nat'>
@@ -14,5 +14,8 @@
       <range start='192.168.100.128' end='192.168.100.254'/>
     </dhcp>
   </ip>
+  <dnsmasq:options>
+    <dnsmasq:option value='log-queries'/>
+  </dnsmasq:options>
 </network>

And then you have to restart network using (scary) destroy and start:

virsh net-destroy default
virsh net-start default

You can also peek content of /var/lib/libvirt/dnsmasq/default.conf if there is your option (log-queries in our case).

After restart you can try on Host:

journalctl -u libvirtd -f

And boot any VM that uses NAT network under LibVirt.

I often use private isolated network in LibVirt to avoid mixing copy of production workload (in test VM) with real production (for example, GitLab CE with active push mirrors would screw mirror targets!). To create private IPv4 network we can mostly follow: https://libvirt.org/formatnetwork.html#isolated-network-config

Examples below was tested on:

• Host: openSUSE LEAP 15.6
• Guest (VM): Ubuntu 24.04 LTS

Here is full definition of LibVirt network named private stored in net-private.xml with contents:

<network xmlns:dnsmasq='http://libvirt.org/schemas/network/dnsmasq/1.0'>
  <name>private</name>
  <bridge name="virbr9"/>
  <domain name='example.com' localOnly='yes'/>
  <dns>
    <host ip='10.99.99.20'>
      <hostname>vm1</hostname>
    </host>
  </dns>
  <ip address="10.99.99.1" netmask="255.255.255.0">
    <dhcp>
      <range start="10.99.99.100" end="10.99.99.200"/>
      <host mac='10:20:30:40:50:60' name='vm1.example.com' ip='10.99.99.20'/>
    </dhcp>
  </ip>
  <ip family="ipv6" address="fd00:0609:c0a8:0a00::1" prefix="64"/>
  <dnsmasq:options>
    <dnsmasq:option value='log-queries'/>
    <dnsmasq:option value='log-facility=/var/log/dnsmasq-private-net.log'/>
  </dnsmasq:options>
</network>

WARNING!

• if you will use above log-facility to log to file you have to prepare log file that is compatible with AppArmor policy - in openSUSE it is in /etc/apparmor.d/usr.sbin.dnsmasq

  /var/log/dnsmasq*.log w,
  

• so we must use log name matching /var/log/dnsmasq*.log

• and set proper permissions:

  sudo chown dnsmasq:nogroup /var/log/dnsmasq-private-net.log
  sudo touch /var/log/dnsmasq-private-net.log

• WARNING! AppArmor will NOT report denied access!

To create, start and autostart this private network at 10.99.99.1/24 you have to run:

xmllint --noout --relaxng /usr/share/libvirt/schemas/network.rng net-private.xml
virsh net-define net-private.xml
virsh net-start private
virsh net-autostart private

To be able to install packages I will also install Proxy under Host (openSUSE LEAP):

sudo zypper in tinyproxy

Next I changed its configuration /etc/tinyproxy/tinyproxy.conf - diff:

23c23
< Port 8888
---
> Port 3128
30c30
< #Listen 192.168.0.1
---
> Listen 10.99.99.1
96c96
< #LogFile "/var/log/tinyproxy/tinyproxy.log"
---
> LogFile "/var/log/tinyproxy/tinyproxy.log"
120c120,121
< LogLevel Info
---
> #LogLevel Info
> LogLevel Connect
190c191
< MaxClients 100
---
> MaxClients 20
201a203
> Allow 10.99.99.0/24
285c287
< #ConnectPort 443
---
> ConnectPort 443

Enable and start tinyproxy with:

sudo systemctl enable --now tinyproxy
# test
netstat -tln | fgrep 3128

  tcp        0      0 10.99.99.1:3128         0.0.0.0:*               LISTEN

# test proxy access - we must use Private network IP, because tinyproxy listens only there
curl -fsS -D - -o /dev/null -x 10.99.99.1:3128 http://www.linux.cz

  HTTP/1.1 200 OK
  Via: 1.1 tinyproxy (tinyproxy/1.11.2)
  Date: Wed, 04 Dec 2024 09:32:28 GMT
  Server: Apache

Now we can create sample VM (I will use Ubuntu 24.04 LTS) - remember to assign it network with name "private".

Once VM boots verify its IP address - must start with 10.99.99. otherwise you are using wrong network:

$ ip -br -4 a | fgrep -w UP

eth0             UP             10.99.99.193/24 metric 100

Step specific for openSUSE LEAP 15.6:

• I have to allow access to port 3128/tcp to zone libvirt using:

firewall-cmd --zone=libvirt --add-port=3128/tcp --permanent
firewall-cmd --reload

• WARNING! Even when I have set firewall-cmd --set-log-denied=unicast there is nothing logged by firewall!

Transparent Proxy access for APT:

• WARNING! Do not set globally proxy variables - there is risk that some system component will misuse it.

• from https://askubuntu.com/a/257296, to enable just APT proxy access, create file /etc/apt/apt.conf.d/99zz_proxy.conf with contents:

  Acquire::http::Proxy "http://10.99.99.1:3128";

Now inside Ubuntu guest you can run sudo apt-get update and ensure that there is NO single line with Ign prefix (Ignored). All should be Get: or Hit:

Now you can try to install some program, for example sudo apt-get install ncdu (neat tool to show directory usages interactively).

To use proxy (temporarily) with other commands I created script /usr/local/bin/http_proxy.sh with contents:

#!/bin/bash
set -euo pipefail
p=10.99.99.1:3128

[ $# -gt 0 ] || {
	echo "Usage: $0 command args ..." >&2
	exit 1
}
set -x
http_proxy=$p https_proxy=$p "$@"
exit 0

Example usage:

http_proxy.sh curl -fsS -D - -o /dev/null https://www.google.com

To enable Virtio GL 3D graphics there is one hidden requirement - service virtnodedevd.service used to enumerate hardware (including GPU cards) on Host. You have to install it with:

zypper in libvirt-daemon-driver-nodedev

And reboot. After reboot verify that following command returns non-empty list and no error:

virsh nodedev-list

Only then you will be able to select your GPU backend (if fulfills additional requirements).

There exist libivrt-nss plugin that allows automatic resolving of running VM names through /etc/nsswitch.conf so you can ping and/or ssh to VM without any trickery in /etc/hosts (or in HostName IP trick in ~/.ssh/config).

To install under Fedora you can just follow: https://libvirt.org/nss.html

dnf install libvirt-nss
authselect enable-feature with-libvirt # this will add "libvirt libvirt_guest" to /etc/nsswitch.conf

But under openSUSE LEAP 15.6 it is more tricky. Installation is similar:

zypper in libvirt-nss

But you have to manually modify /etc/nsswitch.conf because authselect package is incomplete in openSUSE case. So you final line in /etc/nsswitch.conf should look like:

hosts:  	files libvirt libvirt_guest dns

But it is still NOT all. If you now try ssh VM_NAME or ping VM_NAME it will not work:

ssh: Could not resolve hostname VM_NAME: Name or service not known

Audit search will reveal problem:

ausearch -ts boot -m avc -sv no -i -x /usr/sbin/nsc

...
type=PROCTITLE msg=audit(05/19/25 16:31:02.784:3523) : proctitle=/usr/sbin/nscd
type=PATH msg=audit(05/19/25 16:31:02.784:3523) : item=0 \
   name=/var/lib/libvirt/dnsmasq//virbr9.macs inode=3409148 \
   dev=103:02 mode=file,644 ouid=root ogid=root rdev=00:00 \
   nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 \
   cap_frootid=0
type=CWD msg=audit(05/19/25 16:31:02.784:3523) : cwd=/
type=SYSCALL msg=audit(05/19/25 16:31:02.784:3523) : \
  arch=x86_64 syscall=openat success=no \
  exit=EACCES(Permission denied) a0=0xffffff9c \
  a1=0x7fc9ac000f60 a2=O_RDONLY a3=0x0 items=1 ppid=1 \
  pid=5970 auid=unset uid=nscd gid=nscd euid=nscd \
  suid=nscd fsuid=nscd egid=nscd sgid=nscd fsgid=nscd \
  tty=(none) ses=unset comm=nscd exe=/usr/sbin/nscd \
  subj=nscd key=(null)
type=AVC msg=audit(05/19/25 16:31:02.784:3523) : \
  apparmor=DENIED operation=open class="file" profile=nscd \
  name=/var/lib/libvirt/dnsmasq/virbr9.macs pid=5970 comm=nscd \
  requested_mask=r denied_mask=r fsuid=nscd ouid=root

To fix this problem:

• append to /etc/apparmor.d/local/usr.sbin.nscd

  # Site-specific additions and overrides for 'usr.sbin.nscd'
  /var/lib/libvirt/dnsmasq/*.macs r,
  

• reload NSCD profile for AppArmor:

  apparmor_parser -r /etc/apparmor.d/usr.sbin.nscd

Now same command ssh VM_NAME or ping VM_NAME should work without issues.

Encountered on virtio-blk storage.

There is problem when using libvirt on openSUSE LEAP 15.6 (both Linux guests and FreeBSD guests - not guest's fault). Trim command appears to work (even iostat will show discards) but qcow2 file will not shrink.

Solution:

• in virt-manager, go to your Virtio Disk X -> and set Discard mode: unmap
• powercycle VM
• when you do again trim in VM you should see that corresponding qcow2 file will shrink...
• or edit your virsh edit VM XML and add discard='unmap' attribute to corresponding element <driver/>, for example: <driver name='qemu' type='qcow2' discard='unmap'/>

It solved my issue and TRIM works.

As simple indicator - when you invoke TRIM in guest (after this change) you should observe that it is no longer instant - in my case TRIM now takes around 5s to finish. Next verify with ls -lhs file.qcow2 that allocated qcow2 size (number on the left) really shrunk.

To shrink qcow2 completely you need to copy it with qemu-img convert -p -f qcow2 -O qcow2 source.qcow2 copy.qcow2.

Inspired by Qubes I'm looking for firewall per-VM. Most close thing for LibVirt is called nwfilter. Official (incomplete of course) documentation can be found on: https://libvirt.org/formatnwfilter.html

There is a bit hidden hint that nwfilter supports bridges only (but not sure).

So I decided to start with creating main bridge:

• first I renamed LAN card back from enp2s0 to eth0 using:

  grubby --update-kernel=ALL --args=net.ifnames=0
  grub2-mkconfig -o /tmp/grub.cfg
  diff /tmp/grub.cfg /boot/grub2/grub.cfg
  cp /tmp/grub.cfg /boot/grub2/grub.cfg
  # next rename connection
  cd /etc/NetworkManager/system-connections
  mv enp2s0.nmconnection eth0.nmconnection
  vim eth0.nmconnection  # replace your LAN name (enp2s0) with eth0
  # finally
  reboot

• now following: https://dev.to/kubenetic/bridged-networking-on-fedora-workstation-for-virtual-machines-5d51 (changed LAN name)
• WARNING! Do this on LOCAL console - it will break connection until all commands finish:

  nmcli connection delete eth0
  nmcli con add ifname br0 type bridge autoconnect yes con-name br0 ipv4.method auto
  nmcli con add type bridge-slave autoconnect yes con-name br-slave-eth0 ifname eth0 master br0
  nmcli connection up br0

Now it is good time to verify that any VM (say with Alpine Linux) will work - acquire IP address

• in my case it did not work and dmesg quickly revealed problem:

  $ sudo dmesg -t | grep REJECT
  
  filter_IN_FedoraWorkstation_REJECT: IN=br0 OUT= MAC=... SRC=192.168.0.1 DST=192.168.0.100 ... PROTO=UDP SPT=67 DPT=68 LEN=556
  filter_IN_FedoraWorkstation_REJECT: IN=br0 OUT= MAC=... SRC=192.168.0.1 DST=192.168.0.100 ... PROTO=UDP SPT=67 DPT=68 LEN=556

• so we have to add "service" dhcp to active zone:

  $ sudo firewall-cmd --get-active-zones
  
  FedoraWorkstation (default)
    interfaces: br0 eth0
  libvirt
    interfaces: virbr0
  
  $ z=FedoraWorkstation # replace with your active zone
  $ firewall-cmd --info-zone=$z
  
  FedoraWorkstation (default, active)
    target: default
    ingress-priority: 0
    egress-priority: 0
    icmp-block-inversion: no
    interfaces: br0 eth0
    sources:
    services: dhcpv6-client
    ports:
    protocols:
  ...

• please note that "service" is NOT service named in /etc/services but rather XML definition, normally from /usr/lib/firewalld/services/

• we can also dump named service with command like:

  # firewall-cmd --info-service=dhcp --verbose
  dhcp
    summary: DHCP
    description: This allows a DHCP server to accept messages from DHCP clients and relay agents.
    ports: 67/udp
  ... empty fields omitted ...

• so in my case I have to do this:

  $ less /usr/lib/firewalld/services/dhcp.xml # review if this is right service
  $ sudo firewall-cmd --zone=$z --permanent --add-service=dhcp
  $ sudo firewall-cmd --reload
  $ firewall-cmd --info-zone=$z | grep services:
  
    services: dhcp dhcpv6-client

• now reboot you VM (tested Alpine Linux) - it should get assigned DHCPv4 address without any issue.

• you can also run virsh dumpxml VM_NAME and verify that <interface/> element is of type bridge:

   <interface type='bridge'>
      <mac address='52:54:00:xx:xx:xx'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

Now we have to install packages required for nwfilter: Installation under Fedora 43 Beta:

sudo dnf install libvirt-daemon-config-nwfilter libvirt-daemon-driver-nwfilter

Available filters can be listed with:

$ virsh nwfilter-list

 UUID                                   Name
-----------------------------------------------------------------
 0b34e409-a036-4e94-8ae2-fa162c29dbb9   allow-arp
 ad182bce-610b-417f-b0be-b56cce8f8e8d   allow-dhcp
 4f4bcab1-6d59-4ae1-b023-40a4034bfcff   allow-dhcp-server
 12ef1073-8a80-41e2-b3dd-f2dd9072d254   allow-dhcpv6
...

In case of Fedora 43 we can find XML definitions on /usr/share/libvirt/nwfilter. I plan to use as starter /usr/share/libvirt/nwfilter/clean-traffic.xml

Things are a bit more difficult because there are two ways how to use nwfilter:

1. add <filterref filter='NAME'/> right into <interface> element of VM definition
2. use nwfilter-binding-create

According to https://www.petrospetrou.co.uk/virtualization/2024/09/23/libvirt-net-filter.html later option is better because it does not require restart of VM after filter change.

Problem: when I simply run virsh edit VM_NAME and added <filterref filter='clean-traffic'/> under element <interfce/> there was error in log:

 $ journalctl -b -u virtnwfilterd -p 4

 ... fed-apac virtnwfilterd[26271]: encountered an error on interface vnet1 index 6: No such device

To find more I made these changes in /etc/libvirt/virtnwfilterd.conf

log_filters="1:nwfilter"
log_outputs="1:file:/var/log/nwfilter-debug.log"

Right after configuration file write it should automatically restart. When I power-cycled VM I got a bit more detail in ``:

debug : learnIPAddressThread:383 : Couldn't open device vnet7: vnet7: can't mmap rx ring: Permission denied
error : learnIPAddressThread:622 : encountered an error on interface vnet7 index 12: No such device

There is recent issue on: https://www.spinics.net/linux/fedora/libvir/msg251548.html

On Tue, Feb 25, 2025 at 10:30:46AM +0100, Pavel Hrdina wrote:
...
   - on Fedora 41 current selinux policy silently blocks virtnwfilterd
     to create_socket_perms for packet_socket resulting in this error:

       internal error: setup of pcap handle failed: can't mmap rx ring: Permission denied

     I've created temporary selinux module to allow this permission, need
     to post a patch to fedora selinux policy to fix this.

Tried:

# dnf install selinux-policy-doc
# rpm -ql selinux-policy-doc | grep nwfilt

  /usr/share/man/man8/virtnwfilterd_selinux.8.gz
# man 8 virtnwfilterd_selinux

Trying temporary workaround from manual page:

$ sudo semanage permissive -a virtnwfilterd_t

Yup! There is no longer error in /var/log/nwfilter-debug.log

Note that thanks to DHCP-snooping (or "learning") feature there will be automatically created rule for snooped IP address from DHCP server in my case 192.168.0.102:

# ebtables-save | fgrep -e .102 -e 28:5b

-A I-vnet10-arp-ip -p ARP --arp-ip-src 192.168.0.102 -j RETURN
-A I-vnet10-arp-mac -p ARP --arp-mac-src 52:54:00:b2:28:5b -j RETURN
-A I-vnet10-ipv4-ip -p IPv4 --ip-src 192.168.0.102 -j RETURN
-A I-vnet10-mac -s 52:54:00:b2:28:5b -j RETURN
-A I-vnet10-rarp -p RARP -s 52:54:00:b2:28:5b -d Broadcast \
  --arp-op Request_Reverse --arp-ip-src 0.0.0.0 --arp-ip-dst 0.0.0.0 \
  --arp-mac-src 52:54:00:b2:28:5b --arp-mac-dst 52:54:00:b2:28:5b -j ACCEPT
-A O-vnet10-rarp -p RARP -d Broadcast --arp-op Request_Reverse \
  --arp-ip-src 0.0.0.0 --arp-ip-dst 0.0.0.0 --arp-mac-src 52:54:00:b2:28:5b \
  --arp-mac-dst 52:54:00:b2:28:5b -j ACCEPT

Or we can see it globally with nft list ruleset.

That's magic implemented in libvirt-11.6.0/src/nwfilter/nwfilter_learnipaddr.c source.

TODO: Filtering input ports - following

• https://libvirt.org/formatnwfilter.html#second-example-custom-filter

• please note that proper rule version is that one using connection state filter:

  <rule action='accept' direction='in'>
    <tcp dstportstart='80' state='NEW'/>
  </rule>

• one should never use just <rule action='accept' direction='in'><tcp dstportstart='80'/></rule> for TCP, because it allows malicous traffic to port 80, for example, fake RST packets, while previous version is using connection tracking to pass only valid data.