Cloud hypervisor - AmpereComputing/ampere-lts-kernel---DEPRECATED GitHub Wiki

Cloud Hypervisor on Altra

Build cloud-hypervisor

Follow doc: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/building.md

Build dynamic linked binary

cargo build --release

file target/release/cloud-hypervisor
target/release/cloud-hypervisor: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=6188673dff9f9a438cc8bbef34eb769fb3d74575, for GNU/Linux 3.7.0, stripped

Build static linked binary

Need to use musl as C library. The original doc does not work. On ubunut-2004:

rustup target add aarch64-unknown-linux-musl
cargo build --release --target=aarch64-unknown-linux-musl --all
error occurred: Failed to find tool. Is `aarch64-linux-musl-gcc` installed?

Do bellow:

$ sudo apt install musl-tools
$ CC=musl-gcc cargo build --release --target=aarch64-unknown-linux-musl --all

$ file target/aarch64-unknown-linux-musl/release/cloud-hypervisor
target/aarch64-unknown-linux-musl/release/cloud-hypervisor: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=36e2a5f9869eb13fe004323c26794f1cbe2f6842, stripped

It is also possible to use docker build cloud hypervisor, e.g:

$ ./scripts/dev_cli.sh build --release --libc musl
[Cloud Hypervisor] Binaries placed under /home/adam/cloud_hyp/cloud-hypervisor/cloud-hypervisor/build/cargo_target/aarch64-unknown-linux-musl/release

$ size build/cargo_target/aarch64-unknown-linux-musl/release/cloud-hypervisor
   text    data     bss     dec     hex filename
2856088  382716    7168 3245972  318794 build/cargo_target/aarch64-unknown-linux-musl/release/cloud-hypervisor

Run cloud-hypervisor

./target/release/cloud-hypervisor \
          --kernel ../linux-cloud-hypervisor/arch/arm64/boot/Image  \
          --disk path=$ROOTFS path=/tmp/ubuntu-cloudinit.img  \
          --cmdline "console=hvc0 root=/dev/vda1 rw" \
          --cpus boot=4   \
          --memory size=0,shared=on  \
          --memory-zone id=mem0,size=1G,shared=on,host_numa_node=0 \
          --net "tap=,mac=,ip=,mask="

Enable CPU affinity:

/root/workloads/cloud-hypervisor-static-aarch64 \
        --api-socket /tmp/ch0 \
        --event-monitor path=/tmp/event.json \
        --cpus boot=40,affinity=[0@[1],1@[2],2@[3],3@[4],4@[5],5@[6],6@[7],7@[8],8@[9],9@[10],10@[11],11@[12],12@[13],13@[14],14@[15],15@[16],16@[17],17@[18],18@[19],19@[20],20@[21],21@[22],22@[23],23@[24],24@[25],25@[26],26@[27],27@[28],28@[29],29@[30],30@[31],31@[32],32@[33],33@[34],34@[35],35@[36],36@[37],37@[38],38@[39],39@[40]] \
        --memory size=128G \
        --kernel /root/workloads/CLOUDHV_EFI.fd \
        --disk path=/root/images/Fedora-Server-38-1.6.aarch64.raw \
        --disk path=/root/workloads/cloudinit,iommu=on \
        --vsock cid=3,socket=/tmp/vsock \
        --net fd=3,mac=$mac 3<>$"$tapdevice"

Control VM with ch-remote

We can use ch-remote tool to control an running Virtual Machine. See doc: https://www.cloudhypervisor.org/docs/prologue/commands/#ch-remote-binary

Start a VM with --api-socket option

sudo ./cloud-hypervisor ...  --api-socket /tmp/cloud-hypervisor.sock

sudo ./ch-remote --api-socket /tmp/cloud-hypervisor.sock reboot

Firmware boot

Rust Hypervisor Firmware

TBD - https://github.com/cloud-hypervisor/rust-hypervisor-firmware/issues/198, pretty new.

用hypervisor-fw_arm64,配置focal-server-cloudimg-arm64.raw,启动后马上报错
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: /EFI/BOOT/BOOTAA64.EFI

用hypervisor-fw_arm64和bionic-server-cloudimg-arm64.raw,可以成功启动vm
Found EFI partition
Filesystem ready
Error loading default entry: File(NotFound)
Using EFI boot.
Found bootloader: /EFI/BOOT/BOOTAA64.EFI
Executable loaded
error: no suitable video mode found.
error: no such device: root.

Press any key to continue...

uname -a
Linux cloud 4.15.0-106-generic

EDK2 Firmware

CH issue:The latest CLOUDHV_EFI.fd can not start aarch64 VM

TBD - https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/uefi.md#building-uefi-firmware-for-aarch64

使用CLOUDHV_EFI.fd启动VM时卡住,配合--serial tty --console off可以启动GRUB,但启动ubuntu时报错:

Synchronous Exception at 0x000000006B8D5624

Reference cloud-hypervisor/scripts/common-aarch64.sh,build cloudhv_efi.fd with old code,then VM can work:

Reset edk2 to 46b4606ba23498d3d0e66b53e498eb3d5d592586,reset edk2-platforms to 8227e9e9f6a8aefbd772b40138f835121ccb2307,
reset acpica to b9c69f81a05c45611c91ea9cbce8756078d76233.
Rebuild CLOUDHV_EFI.fd.

I have created a issue:https://github.com/cloud-hypervisor/cloud-hypervisor/issues/5202

ssh

Input password automatically,then send a command:

sshpass -p cloud123 ssh [email protected] -o "StrictHostKeyChecking no" "lsblk"

--serial,--console

Debug

Cloud-hypervisor debug

rust-gdb --args ./target/aarch64-unknown-linux-gnu/debug/cloud-hypervisor --cpus boot=2 --memory "size=512M" --kernel "/root/workloads/CLOUDHV_EFI.fd" \
        --cmdline "root=/dev/vda1 console=hvc0 rw systemd.journald.forward_to_console=1" \
        --disk "path=/tmp/osdisk.img" --disk "path=/tmp/cloudinit" --api-socket "/tmp/cloud-hypervisor.sock" --serial tty --console off -v -v

Integration debug

export BUILD_TARGET=aarch64-unknown-linux-gnu
rust-gdb --args target/aarch64-unknown-linux-gnu/debug/deps/integration-ff955109ef0d1525 -- --test-filter test_virtio_iommu

By default the Rust test harness hides output from test execution to keep results readable. Test output can be recovered (e.g., for debugging) by passing --nocapture to the test binaries:

cargo test -- --nocapture

You can add log by println!("{}",command); in ssh_command_ip_with_auth and exec_host_command_status to track the test logic.

Network configuration

Please check Cloud-hypervisor-network-configuration

Performance test

Details:Cloud-Hypervisor Performance Metrics

VMM features

Details:Features-and-functionality

Errors and fix

Details:Errors and Fixes