Checkpoint for bhyve using zfs snapshots and clones - FreeBSD-UPB/freebsd-src GitHub Wiki

To use the checkpoint mechanism for bhyve it is required to:

  • (re)build both the userspace and the kernelspace by following the steps from here

NOTE: Modify the Clone FreeBSD-UPB/freebsd-src project section as follows:

root@host # git clone https://github.com/FreeBSD-UPB/freebsd-src
root@host # cd freebsd-src
root@host # git checkout projects/bhyve_checkpoint_libzfs
  • build and run a daemon that uses functions exposed by the zfs library, libzfs.

NOTE: In order to avoid time consuming copy operations between different filesystems(zfs datasets), symbolic links are used in order to minimize the downtime for the guest as much as possible.

Daemon building and running instructions

Building

In order to build the daemon openzfs repository is needed. For that use the following command as root:

root@freebsd:# git clone https://github.com/openzfs/zfs

NOTE: The path for the zfs repository it can be different from /root.

NOTE: There is no need for an actual install but if there are any problems you can use the following commands in trying to fix them

root@freebsd:# cd zfs
root@freebsd:# ./autogen.sh
root@freebsd:# ./configure
root@freebsd:# gmake -j`sysctl -n hw.ncpu`

After cloning the zfs repository navigate to the location where freebsd-src was cloned. In that directory you will find a directory called zfs_daemon; inside there is a Makefile. The Makefile can receive a parameter called ZFS_PATH which can be used to specify the path for the cloned zfs repository. By default, ZFS_PATH is /root/zfs.

In order to compile the daemon use the following commands:

root@freebsd:# make # or use gmake # for using the default value for ZFS_PATH

or

root@freebsd:# make ZFS_PATH=PATH_TO_ZFS_REPOSITORY # or use gmake ZFS_PATH=PATH_TO_ZFS_REPOSITORY  # for specifying a different path for the zfs repository

Running

For running the zfs_daemon use the following command:

root@freebsd:# make run # or use gmake run

Requirements, restrictions and usage

Requirements for using the proposed checkpoint mechanism

  • a FreeBSD machine that uses the zfs filesystem
  • the files for the guest(disks) should be placed in a directory that has a zfs filesystem mounted in it; the state files that are created should be in the same directory because a zfs dataset is needed to create zfs snapshots and zfs clones

Restrictions for using the proposed checkpoint mechanism

  • the daemon must run before creating a checkpoint; otherwise a guest reboot is needed
  • the checkpoint deletion must have the guest running and a checkpoint that is a part of the subtree of the checkpoint that is currently used cannot be deleted
  • if the daemon is stopped or crashes the list of checkpoints will be lost and there is no way to rollback to a checkpoint(the zfs snapshots and clones will still be created but they will not accessible through the daemon anymore; the files can still be restored manually)
  • when requesting a checkpoint creation, the full name for the zfs dataset along with the full path for the state files must be provided

Usage

Checkpoint creation

Generic command
root@freebsd:# bhyvectl --checkpoint <zfs_dataset_name>@<full_path_to_zfs_dataset_mountpoint>/<checkpoint_file_name> --vm=<guest_name>
Example

For a freebsd guest called freebsd_guest and a zfs dataset mounted at /root/freebsd_guest the checkpoint creation command will look like this:

root@freebsd:# bhyvectl --checkpoint zroot/freebsd_guest_ckp@/root/freebsd_guest_ckp/file.ckp --vm=freebsd_guest

Checkpoint rollback

Generic command
root@freebsd:# bhyve <guest_parameters> -r <zfs_dataset_name>@<full_path_to_zfs_dataset_mountpoint>/<checkpoint_file_name> <guest_name>
Example

For a freebsd guest called freebsd_guest and a zfs dataset mounted at /root/freebsd_guest the checkpoint rollback command will look like this:

root@freebsd:# bhyve -c 1 -m 256M -H -A -P -s 0:0,hostbridge -s 1:0,lpc -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 2:0,virtio-net,tap0 -s 3:0,ahci-hd,/root/freebsd_guest_ckp/freebsd_main.img -l com1,stdio -r zroot/freebsd_guest_ckp@/root/freebsd_guest_ckp/file.ckp freebsd_guest

NOTE: The rollback command is part of the restore process for a guest, this being the exact moment when the clones and symbolic links are created.

Checkpoint deletion

Generic command
root@freebsd:# bhyvectl --delete-checkpoint <zfs_dataset_name>@<full_path_to_zfs_dataset_mountpoint>/<checkpoint_file_name> --vm=<guest_name>
Example

For a freebsd guest called freebsd_guest and a zfs dataset mounted at /root/freebsd_guest the checkpoint deletion command will look like this:

root@freebsd:# bhyvectl --delete-checkpoint zroot/freebsd_guest_ckp@/root/freebsd_guest_ckp/file01.ckp --vm=freebsd_guest

NOTE: The deletion command will delete all the subtree created under the point that is requested for deletion.

Demo

A demo for how the proposed solution works can be seen here.

⚠️ **GitHub.com Fallback** ⚠️