CLH VM network setup

macvtap

Follow doc: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/macvtap-bridge.md

make sure host kernel enabled tap, tun and macvtap:

modprobe tap
modprobe tun

net fd=[fd],mac=[MAC] [fd]<>/dev/tapX

Make sure do NOT run CLH with sudo, otherwise there will be error: Error booting VM: VmBoot(DeviceManager(CreateVirtioNet(TapError(IoctlError(2147767506, Os { code: 25, kind: Uncategorized, message: "Inappropriate ioctl for device" })))))

Refer to: https://github.com/cloud-hypervisor/cloud-hypervisor/issues/4917#issuecomment-1442834740

[setup macvtap on host]
# The MAC address must be attached to the macvtap and be used inside the guest
mac="c2:67:4f:53:29:cb"
# Host network adapter to bridge the guest onto
host_net="enP4p4s0"

# Create the macvtap0 as a new virtual MAC associated with the host network
sudo ip link add link "$host_net" name macvtap0 type macvtap
sudo ip link set macvtap0 address "$mac" up
sudo ip link show macvtap0

# A new character device is created for this interface
tapindex=$(< /sys/class/net/macvtap0/ifindex)
tapdevice="/dev/tap$tapindex"

# Ensure that we can access this device
sudo chown "$UID.$UID" "$tapdevice"

[login as root, not sudo]
./target/release/cloud-hypervisor --kernel ../linux-cloud-hypervisor/arch/arm64/boot/Image --disk path=../focal-server-cloudimg-arm64.raw,iommu=on --disk path=/tmp/ubuntu-cloudinit.img --cmdline 'console=hvc0 root=/dev/vda1 rw' --cpus boot=4 --memory size=0,shared=on --memory-zone id=mem0,size=1G,shared=on,host_numa_node=0 --api-socket /tmp/cloud-hypervisor.sock --net fd=3,mac=c2:67:4f:53:29:cb 3<>/dev/tap9

On VM run dhcp to get ip address:

ip addr
sudo dhclient ens4p0

Hotplug macvtap:

Details in: https://github.com/cloud-hypervisor/cloud-hypervisor/discussions/5084#discussioncomment-4866815 Make sure boot VM with UEFI:

While ACPI requires UEFI booting. That means you must boot the VM from a firmware, not from the direct kernel. This restriction is only for AArch64. On X86, the VM booting from direct kernel can also play ACPI (and hotplug of course).

Setup steps refer to https://github.com/cloud-hypervisor/cloud-hypervisor/issues/4917#issuecomment-1336125939

# suppose host sets up /dev/tap9
sudo ./cloud-hypervisor/cloud-hypervisor/target/release/ch-remote --api-socket /tmp/cloud-hypervisor.sock add-net tap=tap9

Or, refer to CLH integration test case: test_macvtap_hotplug:

./scripts/dev_cli.sh tests --integration -- --test-filter test_macvtap_hotplug

It calls:

docker run --workdir /cloud-hypervisor --rm --privileged --security-opt seccomp=unconfined --ipc=host --net=bridge --mount type=tmpfs,destination=/tmp --volume /dev:/dev --volume /home/dom/cloud-hypervisor/cloud-hypervisor:/cloud-hypervisor --volume /root/workloads:/root/workloads --env USER=root --env CH_LIBC=gnu cloudhypervisor/dev:20230116-0 ./scripts/run_integration_tests_aarch64.sh --hypervisor kvm --test-filter test_macvtap_hotplug

Or invoke the test directly:

USER=root CH_LIBC=gnu ./scripts/run_integration_tests_aarch64.sh --hypervisor kvm --test-filter test_macvtap_hotplug 2>&1 | tee tlog
Or:
cargo test common_parallel::test_macvtap_hotplug --target aarch64-unknown-linux-gnu -- --nocapture 2>&1 | tee tlog_cargo`

The test case actually did bellow setup:

command = sudo ip link delete guestmacvtap1
command = sudo ip link delete hostmacvtap1
command = sudo ip link add link enP9p3s0 name guestmacvtap1 type macvtap mod bridge
command = sudo ip link set guestmacvtap1 address 12:34:56:78:90:01 up
command = sudo ip link show guestmacvtap1
command = sudo chown $UID.$UID /dev/tap107
command = sudo ip link add link enP9p3s0 name hostmacvtap1 type macvtap mod bridge
command = sudo ip address add 192.168.1.1/24 dev hostmacvtap1
command = sudo ip link set dev hostmacvtap1 up

target/aarch64-unknown-linux-gnu/debug/cloud-hypervisor" "--cpus" "boot=2" "--memory" "size=512M" "--kernel" "/root/workloads/CLOUDHV_EFI.fd" "--cmdline" "root=/dev/vda1 console=hvc0 rw systemd.journald.forward_to_console=1" "--disk" "path=/tmp/chtLUHf9/osdisk.img" "--disk" "path=/tmp/chtLUHf9/cloudinit" "--api-socket" "/tmp/chtLUHf9/cloud-hypervisor.sock" "-v" "-v"

Hotplug network device with ch-remote:

ch-remote /tmp/chtLUHf9/cloud-hypervisor.sock add-net fd=[3,4],mac=12:34:56:78:90:01,num_queues=4
guest:remote_command_w_output,{"id":"_net0","bdf":"0000:00:05.0"}
guest:wait_vm_boot
guest:ssh_command ip -o link
child kill
command = sudo ip link del guestmacvtap1
command = sudo ip link del hostmacvtap1

Use network bridge instead of macvtap

Reference https://github.com/cloud-hypervisor/cloud-hypervisor/issues/4917

On host:
sudo ip tuntap add mytap0 mode tap              #Create tap
sudo brctl addbr br0                            #Create bridge
sudo brctl addif br0 mytap0 enP9p3s0            #Connect tap and netcard with bridge
sudo ifconfig mytap0 0 up                       #Clear IP of tap
sudo ifconfig enP9p3s0 0 up                     #Clear IP of netcard
sudo dhclient br0                               #Get IP for bridge
sudo dhclient enP9p3s0                          #Get IP for netcard

Run cloud-hypervisor with --net "tap=mytap0"

On VM:
sudo dhclient enp0s5                            #You can find enp0s5 after run "ip addr"