NOTE: The project is under development and may not be completed.

Setup - Clone, build, install

Clone project

root@host # git clone https://github.com/FreeBSD-UPB/freebsd-src
root@host # cd freebsd

Note: The project should be built and installed on both source and destination hosts. Alternatively, the project can be cloned on the shared storage (using NFS as described in the following step) then built and installed from there on each system.

Choose branch

root@host # git checkout projects/bhyve_live_migration 

Build and install world

root@host # make world -DWITH_BHYVE_SNAPSHOT -j`sysctl -n hw.ncpu`
root@host # reboot

Build and install kernel

# Make sure the BHYVE_SNAPSHOT option is present in the GENERIC file.
root@host # egrep "^options\s+BHYVE_SNAPSHOT" sys/amd64/conf/GENERIC

# Add it:
root@host # echo "options BHYVE_SNAPSHOT" >> sys/amd64/conf/GENERIC

root@host # make kernel -j`sysctl -n hw.ncpu`
root@host # reboot

Setup - Shared Storage using NFS (Network File System)

To work, the migration process needs both (source and destination) hosts to share the same disk image. You can set up a shared storage using NFS following the steps described here. You can set up one of the hosts to be the server and the other one to be the client.

Setup - Create a FreeBSD virtual machine using bhyve

To create a virtual machine using bhyve, you can follow the tutorial from here.

NOTE: It is mandatory for the created guest.img to be shared between the two host systems.

Migration Usage

Run the virtual machine on the source host

Run the previously created virtual machine using bhyve.

# start the virtual machine
root@source_host # bhyve -c 1 -m 1024M -H -A -P -s 0:0,hostbridge -s 1:0,lpc -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 3:0,ahci,guest.img -l com1,stdio guest_vm 
....
root@guest #

Destination host migration process

In order to migrate a virtual machine (be it warm or live), an empty virtual machine should be started on the destination. It will wait for a connection from the specified host on the specified port.

To start the virtual machine for migration on the destination host, the -R host[:port] option should be added to the bhyve command. The default port is 24983 (from "bhyve").

# start the virtual machine for migration
# note the -R host[:port] option
root@destination_host # bhyve -c 1 -m 1024M -H -A -P -s 0:0,hostbridge -s 1:0,lpc -s 3:0,ahci,guest.img -l com1,stdio -R source_host:24983 guest_vm_dst

Source host migration process

To migrate a running virtual machine, run bhyvectl with either --migrate=host[:port] for warm migration or --migrate-live=host[:port] for live migration. The flags are mutually exclusive.

Warm migration

root@source_host # bhyvectl --migrate=destination_host:24983 --vm=guest_vm

Live migration

root@source_host # bhyvectl --migrate-live=destination_host:24983 --vm=guest_vm

On the source host, the virtual machine will be closed and destroyed. On the destination host, after the migration process is completed, the virtual machine will resume execution from where it left off.