Kernel compartmentalization - CTSRD-CHERI/cheripedia GitHub Wiki

This page describes how to get started with kernel compartmentalization (kernel-c18n) in CheriBSD. Currently, the implementation of this feature supports only Arm Morello and is being developed in the kernel-c18n branch of CheriBSD. You also need to use the kernel-c18n branch of CHERI LLVM for Morello to build the kernel-c18n CheriBSD branch.

We use the separate CHERI LLVM for Morello branch to make sure it stays compatible with the kernel implementation. Historically it differed from the morello/dev branch and currently its main purpose is to track the c18n_policy branch that is used in the user-space sub-library compartmentalization.

Setup

You can run the kernel-c18n branch natively on Arm Morello using the Type-2 hypervisor bhyve or by emulating it under QEMU either on Arm Morello or other platforms. To make modifications in how the kernel should be compartmentalized, you have to prepare a work environment that lets you rebuild a disk image for a VM. In the future, kernel-c18n will be included in CheriBSD releases and users will be able to either select the compartmentalized kernel on boot or make it default in their setup.

Building a disk image

The process of building a disk image is the same regardless of your host platform. It can be CheriBSD but also Linux or macOS. In order to build kernel-c18n for the first time:

  • Clone cheribuild.py:

    git clone https://github.com/CTSRD-CHERI/cheribuild

  • Follow pre-build setup steps of cheribuild.py

  • Compile the branches of CHERI LLVM for Morello and CheriBSD, and build a disk image for use with bhyve or QEMU:

    ./cheribuild.py disk-image-morello-purecap \
  --include-dependencies \
  --source-root ~/cheri-kernel-c18n \
  --cheribsd-morello-purecap/kernel-config GENERIC-MORELLO-PURECAP-COMPARTMENTS \
  --cheribsd-morello-purecap/git-revision origin/kernel-c18n \
  --morello-llvm/git-revision origin/kernel-c18n

The above command uses a custom source root directory (--source-root) not to conflict with other cheribuild workflows as it uses a custom LLVM branch. cheribuild allows to specify custom build directories for CheriBSD and other projects but not for LLVM. By using a single source root directory for all cheribuild workflows, we would be forced to use the kernel-c18n branch of LLVM for all environments.

When building CheriBSD and its disk image, the above command makes the kernel GENERIC-MORELLO-PURECAP-COMPARTMENTS the default one. This kernel implements kernel-c18n on top of GENERIC-MORELLO-PURECAP that you would in dev or release branches of CheriBSD. In addition to kernel-c18n support, it disables some features not to include the zlib compression library in the kernel itself but allow a user to dynamically load the zlib kernel module.

You can find the explanation on cheribuild's flags in the output of:

./cheribuild.py --help

Running a VM with bhyve

You can run kernel-c18n natively under bhyve on CheriBSD/Morello. It is much faster than QEMU and hence much more useful when testing rapid changes. To do that:

  • Install the cheri-vm-support package:

    sudo pkg64c install cheri-vm-support

    The package is pre-installed on CheriBSD/Morello if you selected an option to do so in the CheriBSD installer.

  • Run bhyve with a path to your disk image:

    sudo bhyve \
  -m 1G \
  -c 1 \
  -s 0,hostbridge \
  -s 1,virtio-blk,~/cheri-kernel-c18n/output/cheribsd-morello-purecap.img \
  -o bootrom=/usr/local/share/u-boot/u-boot-bhyve-arm64/u-boot.bin \
  -o console=stdio \
  kernel-c18n

    You can find the explanation on the above flags in man 8 bhyve (or at man.CheriBSD.org).

In order to stop your VM, use:

sudo sysctl hw.vmm.destroy=kernel-c18n

Running a VM with QEMU

You can emulate kernel-c18n with QEMU on CheriBSD/Morello and other platforms. Despite of being much slower than bhyve, QEMU is still very useful when debugging a kernel. For example, you can configure QEMU to collect information on executed instructions and CPU state transitions. When used in conjuction with QEMU monitor and GDB, you can collect such traces between arbitrary breakpoints.

On CheriBSD/Morello, you can install QEMU with pkg64 install qemu-cheri. However, given the experimental nature of kernel-c18n, we recommend to build the latest revision of QEMU with cheribuild:

./cheribuild.py qemu --source-root ~/cheri-kernel-c18n

Then, you can also use cheribuild to run a VM:

  ./cheribuild.py run-morello-purecap --source-root ~/cheri-kernel-c18n

Incremental changes

If you make changes to the kernel and nothing else, you can quickly rebuild a disk image with:

./cheribuild.py cheribsd-morello-purecap disk-image-morello-purecap \
  --source-root ~/cheri-kernel-c18n \
  --skip-update \
  --skip-world \
  --cheribsd/fast \
  --cheribsd-morello-purecap/kernel-config GENERIC-MORELLO-PURECAP-COMPARTMENTS

The above command:

  • Specifies targets (cheribsd-morello-purecap disk-image-morello-purecap) and does not build their dependencies (no --include-dependencies)

  • Skips updating repositories (--skip-update)

  • Skips building userland (--skip-world)

  • Does not clean previously built object files (--cheribsd/fast)

We recommend making use of a JSON file to configure cheribuild.py rather than specifying its arguments each time you execute the script. To do that, create a symbolic link:

ln -s /path/to/cheribuild.py kernel-c18n-cheribuild.py

and paste into ~/.config/kernel-c18n-cheribuild.json:

{
	"source-root": "~/cheri-kernel-c18n",
	"skip-update": true,
	"skip-world": true,
	"cheribsd": {
		"fast": true
	},
	"cheribsd-morello-purecap": {
		"kernel-config": "GENERIC-MORELLO-PURECAP-COMPARTMENTS",
		"git-revision": "origin/kernel-c18n"
	},
	"morello-llvm": {
		"git-revision": "origin/kernel-c18n"
	}
}

From now on, you can simply execute ./kernel-c18n-cheribuild.py disk-image-morello-purecap to rebuild a disk image. If you decide to use the above configuration, remember to regularly update and rebuild your local clones of repositories as the configuration skips updating them.

You can also consider replacing disk-image-morello-purecap with disk-image-minimal-morello-purecap to build a disk image that would run in the single-user mode and skip running most of startup scripts. Note that the minimal disk image does not include all kernel modules.

Debugging

You can attach GDB to a bhyve or a QEMU process and interrupt the execution of a kernel to investigate the state of compartments.

Preparing GDB

  • Compile GDB using the cheri-14-kgdb branch:

    ./cheribuild.py gdb --source-root ~/cheri-kernel-c18n

  • Create a GDB configuration file. For example in ~/kernel-c18n-gdbinit.txt:

    set $source_root="/path/to/cheri-kernel-c18n"

target remote localhost:1234
eval "set sysroot %s/output/rootfs-morello-purecap", $source_root
eval "set substitute-path %s/output/rootfs-morello-purecap /", $source_root
eval "set debug-file-directory %s/output/rootfs-morello-purecap/usr/lib/debug", $source_root
eval "file %s/output/rootfs-morello-purecap/boot/kernel/kernel.full", $source_root

    You have to set $source_root to an absolute path to your cheri-kernel-c18n directory.

Running bhyve with GDB

  • Run bhyve(8) with the -G flag to specify a GDB server port (1234 in this case):

    sudo bhyve \
  -G 1234 \
  -m 1G \
  -c 1 \
  -s 0,hostbridge \
  -s 1,virtio-blk,~/cheri-kernel-c18n/output/cheribsd-morello-purecap.img \
  -o bootrom=/usr/local/share/u-boot/u-boot-bhyve-arm64/u-boot.bin \
  -o console=stdio \
  kernel-c18n

  • Run gdb with the above configuration:

    ~/cheri-kernel-c18n/output/sdk/bin/gdb -x ~/kernel-c18n-gdbinit.txt

Running QEMU with GDB

  • Run cheribuild with additional flags to wait for a GDB client before starting a VM:

    ./cheribuild.py run-morello-purecap \
  --source-root ~/cheri-kernel-c18n \
  --wait-for-debugger

  • Run gdb with the above configuration:

    ~/cheri-kernel-c18n/output/sdk/bin/gdb -x ~/kernel-c18n-gdbinit.txt

Limitations

The current implementation of kernel-c18n is a prototype of a compartmentalization model that we envision to be included in the dev branch of CheriBSD. It is sufficient to show the benefits of kernel-c18n but it is not mature enough and lacks many features that users and developers would expect.

The current limitations include:

  • The kernel supports only Arm Morello

    We plan to add CHERI-RISC-V support.

  • The kernel runs only under bhyve and QEMU

    You cannot run it directly on Arm Morello hardware due to missing SMP support. We plan to extend the implementation to allow that, e.g. to run the GUI stack with the user-space library-based compartmentalization and the kernel compartmentalization.

  • Debuggers do not unwind stacks

    ddb and GDB debuggers cannot unwind a stack with call frames across compartments.

  • The build system does not implement an infrastructure to compose policies

    The kernel includes a hardcoded policy that is specific to GENERIC-MORELLO-PURECAP-COMPARTMENTS. Instead, we would like the build system to use JSON policy files corresponding to options specified in a kernel configuration.