oVirt 4 - hpaluch/hpaluch.github.io GitHub Wiki

oVirt

oVirt is an open-source distributed virtualization solution, designed to manage your entire enterprise infrastructure. oVirt uses the trusted KVM hypervisor and is built upon several other community projects, including libvirt, Gluster, PatternFly, and Ansible.

WARNING! This page is in flux. As of Mar 2024 I'm trying to install fresh oVirt 4.5/EL8 on old Opteron G2 machine. There are some rough edges, because RedHat is pushing EL9 with its x86-64-v2 architecture requirement (EL9 no longer boots on this Opteron machine - glibc simply aborts on such machine killing init process).

See

https://www.phoronix.com/news/RHEL-9-x86-64-v2-Plans

https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level

If you are familiar with VMware you can easily map oVirt terms to VMware:

oVirt Node <=> vSphere - server that can runs virtual machines
Hosted Engine (HE) <=> vCenter as VM on vSphere - central management software - may run on dedicated server or as VM inside selected Node(s)
oVirt uses Datacenter and Cluster hierarchy
oVirt requires shared storage, any of:
- NFS
- iSCSI
- Gluster
- FCP (Fibre Channel Protocol - for SAN)
NOTE: Unlike vSphere - oVirt 4.5 simply refuses to install and/or use Local storage (although under hood, staged HostedEngineLocal actually run on local storage and LibVirt NAT network for customization before final deployment to shared storage and defined IP address).

Originally oVirt was base product for RHEVM (commercial virtualization sold by RedHat) - similarly as Fedora (or now CentOS stream?) is base for RHEL. However RedHat later abandoned RHEV development as can be seen on https://blogs.ovirt.org/2022/02/future-of-ovirt-february-2022/. Basically some incompetent RedHat/IBM manager decided that only right future is in OpenShift containers and nothing else...

Fortunately it seems that oVirt is still in good shape, because it is also backed by other entities that use it and contribute.

main GitHub namespace: https://github.com/oVirt

Here is my personal top-list of "complexity":

OpenStack - hardest to install - if you can install OpenStack you can do everything in your life! See my project https://github.com/hpaluch/osfs for example - it took me several weeks to install basic OpenStack on bridged network
oVirt - somewhere between - easier to install than OpenStack (also thanks to Ansible). However you still have to carefully plan deployment - including preparation of suitable shared storage
Proxmox VE - very easy to install, plus native support for local storage. However cluster is a bit more fragile and you need to understand what is happening in background. Also there are some poor design decisions - for example dumping and restoring iptables every 10s which put high strain on machine without good reason. Or many commands have arbitrary 10s timeout after witch blind error is reported causing inconsistency (it happened to me when I enlarged virtual disk on loaded machine - UI throw error although command completed successfully later but UI still reported old disk size)

2024 Update

I'm unable to use EL9 version of oVirt 4.5, for example:

https://mirror.rackspace.com/ovirt/ovirt-4.5/iso/ovirt-node-ng-installer/ovirt-node-ng-installer-latest-el9.iso Because it will panic in glibc with this dreadful message:
Run /init as init process
Fatal ...  glibc error: CPU does not ... support x86-64-v2
Ah, yes, that damn x86-64-v2. CPU is old dual-core Opteron.

(Tested under Proxmox VE - to see full error messages - adde HW -> Serial port, on PVE host run 'qm terminal VMID', and in Boot menu I pressed TAB and added console=ttyS0 to boot arguments)

So far, using EL8 Node ISO seems to work: https://mirror.rackspace.com/ovirt/ovirt-4.5/iso/ovirt-node-ng-installer/ovirt-node-ng-installer-latest-el8.iso
details on OS version: CentOS Stream: 8.7.2206.0

kernel 4.18.0-526.el8.x86_64, rpm -q kernel => `kernel-4.18.0-526.el8.x86_64``

in case of bare-metal I will have to rebuild somewhere:

https://elrepo.org/linux/elrepo/el8/SRPMS/kmod-forcedeth-0.0-9.el8_7.elrepo.src.rpm

in above example ovirt45-sh.example.com is our main "Node" FQDN and manage.example.com will be used for "Engine Appliance" - VM deployed to "Node"

you may login as root even via SSH (it is enabled in "Node").
added few useful packages:
dnf install --repo baseos net-tools # e.g. netstat
dnf install --repo appstream mc vim-enhanced
before installing "Engine" Appliance (VM) we have to install it in "Node" with:
# command below will download 1.8GB (Engine VM)
dnf install ovirt-engine-appliance
# there was GPG key problem, tried quick workaround:
rpm -ivh /var/cache/dnf/ovirt-45-upstream-62781357e04a3932/packages/ovirt-engine-appliance-4.5-20231201120252.1.el8.x86_64.rpm
after above install I strongly recommend to snapshot "Node" - so far no oVirt was configured yet
when ready - main deployemnt ot "Engine" is done with command (--4 forces IPv4 stack):
hosted-engine --deploy --4

How to get working NVidia LAN - forcedeth module is missing from Node install (EL8), because upstream removed nearly all useful modules several years ago:

https://linux.web.cern.ch/rhel/releasenotes/RELEASE-NOTES-7.0-x86_64/
so I first installed node rom `ovirt-node-ng-installer-latest-el8.iso`` under KVM

now master art - how to install forcedeth module from: https://sigs.centos.org/kmods/install/

$ dnf install --repo extras centos-release-kmods
$ dnf search --repo centos-kmods forcedeth

CentOS Stream 8 - Kmods                                                                 437 kB/s | 152 kB     00:00    
========================================== Name & Summary Matched: forcedeth ===========================================
kmod-forcedeth.x86_64 : Nvidia nFore Ethernet Controller (forcedeth) driver


$ dnf install --repo baseos --repo centos-kmods kmod-forcedeth

Problem: package kmod-forcedeth-1:4.18.0~539-1.el8s.x86_64 from centos-kmods requires kernel-modules-uname-r >= 4.18.0-539.el8.x86_64, but none of the providers can be installed
- cannot install the best candidate for the job
- package kernel-modules-4.18.0-539.el8.x86_64 from baseos is filtered out by exclude filtering
- package kernel-modules-4.18.0-540.el8.x86_64 from baseos is filtered out by exclude filtering
- package kernel-modules-4.18.0-544.el8.x86_64 from baseos is filtered out by exclude filtering
- package kernel-modules-4.18.0-546.el8.x86_64 from baseos is filtered out by exclude filtering

(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

- Here is the trick - updated versions on locked packages:
```shell
x=/etc/dnf/plugins/versionlock.list
cp $x /root/
# update locks to our preferred kernel :-)
fgrep kernel $x

   kernel-core-4.18.0-526.el8.x86_64
   kernel-tools-libs-4.18.0-526.el8.x86_64
   kernel-4.18.0-526.el8.x86_64
   kernel-tools-4.18.0-526.el8.x86_64
   kernel-modules-4.18.0-526.el8.x86_64

sed  -i.bak  -e 's/4\.18\.0-526/4.18.0-546/' $x

# and try again:

dnf install --repo baseos --repo centos-kmods kmod-forcedeth

# Should now proceed...

reboot

# verify that new kernel was booted and dry-run try to insert module

$ uname -r

4.18.0-546.el8.x86_64

$ modprobe forcedeth

# no error => OK

Cloning VM to real disk:

you may NOT use CloneZilla

why? because there is used LVM Thin:

lvs -a

...
[pool00_tdata]                     onn Twi-ao---- <75.02g
[pool00_tmeta]                     onn ewi-ao----   1.00g
...

solution: use plain DD command (yes it is inefficient, but should work...)

WARNING!

If you clone installed OS HDD with dd there is one serious problem affecting all LVM setups(!):

All LVM commands (pvs, vgs, lvs) will fail because so called PVID of disk changed

Workaround is described here https://bugzilla.redhat.com/show_bug.cgi?id=1969408#c1

just delete file with rm /etc/lvm/devices/system.devices and reboot

after reboot - when you login - there should be no more fatal Python error and also all LVM commands should work again.

Additionally to fixing LVM PVID you have to also fix network card name.

run ip l to see network card names
rename file /etc/sysconfig/network-scripts/ifcfg-enpXXX to your new network card name
additionally you have to update network card name inside ifcfg-enpXXX - both NAME= and DEVICE=
reboot and watch if network was properly configured

Last update:

unfortunately my favorite dual-core Opteron is not liked by oVirt 4.5/EL8:

[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg":
"The host has been set in non_operational status, deployment errors: code 156: Host ovirt-x2.example.com moved to Non-Operational state as host CPU type is not supported in this cluster compatibility version or is not supported at all, code 9000: Failed to verify Power Management configuration for Host ovirt-x2.example.com., fix accordingly and re-deploy."}

- how to find AMD CPU generation:

```shell
virsh -r capabilities | xmllint --xpath '//host/cpu/model/text()' - && echo

Output: Opteron_G2
Full name from /proc/cpuinfo: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+

to study:
- https://lists.ovirt.org/archives/list/[email protected]/thread/NOLWZUDPZMEAJYMNOV75QKQ42BQRQV4B/
- ServerCPUList on: https://raw.githubusercontent.com/oVirt/ovirt-engine/master/packaging/dbscripts/upgrade/pre_upgrade/0000_config.sql
- cluster version 4.2 should be OK.
- https://bugzilla.redhat.com/show_bug.cgi?id=1893387
will try:

hosted-engine --deploy --4 --ansible-extra-vars=he_cluster_comp_version=4.2

oVirt consists of:

one Engine node that contains all management software and web portal and database
Nodes - there are run virtual machines (VMs). Nodes are managed primarily using daemons called VDSM. There are two VDSM daemons for security reasons:
- vdsmd.service is main unprivileged VSDM
- supervdsmd.service is running as root and is called by VDSM for privileged tasks (mounting filesystems etc.)
Recommended: shared storage (NFS, iSCSI, FC...) for ISOs and VMs. Shared storage is the only options supported in clusters. But for single host you can still use Local storage with some gotchas...

Local storage requirements:

be sure to have at least 2 empty disk partitions formatted to supported filesystem - I used ext4
- partition use as ISO type for installation isos
- partition use as Data type for VM data (is it correct?)

My sample setup is:

ovirt-mng.example.com - oVirt Engine. Run as VM in VirtualBox. Installed in top of CentOS 7.4, 1x CPU, 1.8GB RAM (because of host PC constraints), 8GB swap, 60GB VirtualDisk (currently used ~6GB). Installed using guide on https://www.ovirt.org/documentation/quickstart/quickstart-guide/#install-ovirt-engine
ovirt-node1.example.com - oVirt Node with local storage. Installed on bare metal AMD X2, 6GB RAM, 200GB SATA disk. Again CentOS 7.4 (with EPEL's forcedeth network driver - see Missing-NVidia-Network-driver). Prepared using guide on https://www.ovirt.org/documentation/install-guide/chap-Enterprise_Linux_Hosts/ I used following partition layout - GPT:
- biosdev boot 1MB (compatibility layer to allow old MBR based BIOS to boot from GPT disk)
- /boot, ext4, 1GB
- LVM, swap, 8GB
- LVM, /, 50GB for node installation
- LVM, /mnt/vms, 90GB for future VMS (on storage setup this partition must be un-mounted and removed from /etc/fstab)
- LVM, /mnt/isos, 40GB for future ISO library (on storage setup this partition must be un-mounted and removed from /etc/fstab)

Timekeeping:

please ensure that all your hosts have proper time
you can set Local system clock in CentOS 7 using:
```
timedatectl set-local-rtc true
```

Setup outline:

oVirt Engine install:
- installed CentOS 7.4 for oVirt Engine in VirtualBox
- added host entries to /etc/hosts, double checked hostnames, using hostnamectl set-hostname for fix
- disabled setlinux in /etc/sysconfig/selinux
- prepare for setup:
```
yum update -y
init 6 # rebooting new kernel
yum install http://resources.ovirt.org/pub/yum-repo/ovirt-release42.rpm
yum -y install ovirt-engine
```
- recommended: Snapshot of VM, or even export - for easy rebuild
- run engine-setup to create oVirt portal
oVirt Node install
- installed CentOS 7.4 for oVirt Engine on bare-metal Hardware
- added host entries to /etc/hosts, double checked hostnames, using hostnamectl set-hostname for fix
- disabled setlinux in /etc/sysconfig/selinux
- preparing for host addition:
```
yum update -y
init 6 # rebooting new kernel
yum install http://resources.ovirt.org/pub/yum-repo/ovirt-release42.rpm
```
Adding my new Host ovirt-node1.example.com in oVirt portal
- TIP: You can go to Events menu in oVirt portal to to see detailed progress of adding New Host
Creating two basic storages (Warning! Data must be created first! It's the only option in beginning):
Create Local Data storage on Host ovirt-node1: I did following preparation on ovirt-node1:
```
df /mnt/vms # note mount point
   /dev/mapper/centos_ovirt--node1-mnt_vms
chown vdsm:kvm /mnt/vms/
umount /mnt/vms
vi /etc/fstab # comment out line with `/mnt/vms`
```
In oVirt portal specify following on storage creation:
- Storage type: POSIX compliant FS
- Path: /dev/mapper/centos_ovirt--node1-mnt_vms
- VFS Type: ext4
- Name: node1-vms
- Try OK - oh now it works!
- Wait until your storage become Active

Create Local ISO storage on Host ovirt-node1 - similar setup:

df /mnt/isos # note mount point
   /dev/mapper/centos_ovirt--node1-mnt_isos ...
chown vdsm:kvm /mnt/isos/
umount /mnt/isos
vi /etc/fstab # comment out line with `/mnt/isos`
In _oVirt portal_ specify following on storage creation:
- Domain Function: `ISO` - do not forget this!!!
- Storage type: `POSIX compliant FS`
- Path: `/dev/mapper/centos_ovirt--node1-mnt_isos`
- VFS Type: `ext4`
- Name: `node1-isos`
* Try OK - oh now it works!
* Wait until your storage become _Active_

Upload CentOS 7.4 install ISO to Local ISO storage - because of bugs bellow use directly SCP - upload your CentOS ISO (for example CentOS-7-x86_64-Minimal-1708.iso) to /rhev/data-center/mnt/_dev_mapper_centos__ovirt--node1-mnt__isos/fcad81a3-e9fa-4730-903c-6b9be46ee31b/images/11111111-1111-1111-1111-111111111111/ on ovirt-node1. (Warning! you path may differ a bit)
verify that install ISO is visible in portal:
- click on Storage -> Domain
- click on your ISO domain (in my case node1-isos
- click on Images
- Voilà! I can see my CentOS iso there.
Create first VM with CentOS 7.4 - in portal:
- click on Compute -> Virtual Machines
- click on New
- fill in at least:
  - Operating system: RedHat Enteprise Linux 7.x (x64)
  - Optimized for: Server
  - Name: centos7 (required field)
  - nic1: select the only options (in my case ovirtmgmt/ovirtmgmt)
  - Instance Images click on Create:
    - Size(GB): enter 10 for example.
    - Alias: centos-root (required field)
    - click on OK
  - click on Show Advanced Options
  - click on Boot options
  - check on Attach CD
  - ensure that your install ISO is selected
  - select CD-ROM as Second Device (default is empty)
- confirm OK
once your VM is in state Down you can click on Run
wait until your VM is in state Active
then click on Console
if your browser asks you to select program to run *.vv file than you need to download virt-viewer.exe for Windows. Beware - in my case worked only 32-bit version virt-viewer-x86-6.0.msi (64-bit crashed) Done.

Bugs

Data Center "Default" is Down

After reboot of both ovirt-mng and ovirt-node I found that my Default datacenter is down forever. After a while I had found dat datacenter Default has:

Storage Type: Shared

Which is wrong (I use Local storage type). So I fix it following way:

highlight my Default data-center in Compute -> Data Centers
click on Edit
change Storage Type to Local again.
click on OK After a while my data center was _Up_again.

VM or host Nor responding

Sometimes all VMs in host are in state Not responding.

workaround - restart VDSM using:
```
systemctl restart vdsmd
```

QEMU guest agent not responding

When you click on details of your VM you see that all field of Guest Info tab are Unknown even when Guest Agent is running.

Fix: N/A

Tips

Exporting VM to local disk

On your oVirt Node (ovirt-node1 in my example) create directory for export:

mkdir /vm-exports
chown vdsm:kvm /vm-exports/ # probably not needed (it seems to be created by root)

In oVirt portal do this:

highlight your VM for export
click on vertical ... for additional menu
click on Export as OVA
fill in Directory: /vm-exports
click on OK
click on menu Events

Wait until this message appears:

Vm VM_NAME was exported successfully as a Virtual Appliance to path /vm-exports/VM_NAME.ova ...

You can then transfer your VM to client PC using regular SCP/WinSCP program.

Using CLI

On your host (ovirt-node1 in my case) I can use commands (see https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client/):

vdsm-client Host getVMList
  [
    "bcc2845e-f372-4260-8874-7b45bd55f8e9"
  ]

vdsm-client VM getInfo vmID=bcc2845e-f372-4260-8874-7b45bd55f8e9
# lot of output in JSON format

# getting list of commands
 vdsm-client VM help
usage: vdsm-client VM [-h] method [arg=value] ...
vdsm-client VM: error: argument method [arg=value]: invalid choice: 'help' (choose from 'changeFloppy', 'thaw', 'changeCD', 'desktopSendHcCommand', 'setNumberOfCpus', 'getInfo', 'migrate', 'merge', 'diskSizeExtend', 'seal', 'migrationCreate', 'getStats', 'hibernate', 'hostdevHotplug', 'desktopLogin', 'setIoTune', 'desktopLogoff', 'cont', 'hotplugMemory', 'setCpuTunePeriod', 'diskReplicateFinish', 'migrateChangeParams', 'hostdevHotunplug', 'shutdown', 'hotplugDisk', 'getIoTunePolicy', 'getIoTune', 'getMigrationStatus', 'destroy', 'hotplugNic', 'hotunplugLease', 'pause', 'setDestroyOnReboot', 'hotplugLease', 'hotunplugMemory', 'setTicket', 'diskReplicateStart', 'desktopLock', 'updateDevice', 'setCpuTuneQuota', 'create', 'freeze', 'reset', 'updateVmPolicy', 'migrateCancel', 'getDiskAlignment', 'setBalloonTarget', 'hotunplugDisk', 'snapshot', 'hotunplugNic')

vdsm-client VM getStats vmID=bcc2845e-f372-4260-8874-7b45bd55f8e9
# lot of outpout - but does not see guest...