Virtual machine network configuration - AmpereComputing/ampere-lts-kernel---DEPRECATED GitHub Wiki
We can follow macvtap-bridge to configure network for VM.
# The MAC address must be attached to the macvtap and be used inside the guest
mac="c2:67:4f:53:29:cb"
# Host network adapter to bridge the guest onto
host_net="eno1"
# Create the macvtap0 as a new virtual MAC associated with the host network
sudo ip link add link "$host_net" name macvtap0 type macvtap
sudo ip link set macvtap0 address "$mac" up
sudo ip link show macvtap0
# A new character device is created for this interface
tapindex=$(< /sys/class/net/macvtap0/ifindex)
tapdevice="/dev/tap$tapindex"
# Ensure that we can access this device
sudo chown "$UID.$UID" "$tapdevice"
# Use --net fd=3 to point to fd 3 which the shell has opened to point to the /dev/tapN device
target/debug/cloud-hypervisor \
--kernel ~/src/linux/vmlinux \
--disk path=~/workloads/focal.raw \
--cpus boot=1 --memory size=512M \
--cmdline "root=/dev/vda1 console=hvc0" \
--net fd=3,mac=$mac 3<>$"$tapdevice"
To make '3<>/dev/tapX' work, don't run target/debug/cloud-hypervisor with sudo.
If the VM can't browse internet,please try dhclient in VM.
On host:
ifconfig
macvtap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::c067:4fff:fe53:29cb prefixlen 64 scopeid 0x20<link>
ether c2:67:4f:53:29:cb txqueuelen 500 (Ethernet)
RX packets 30344 bytes 2616872 (2.6 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 126 bytes 11911 (11.9 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
+-----------------------------------------------------------------------------------------+
| |
| |
| |
| +---------------------------------+ |
| | | |
| | VM | |
| | | |
| | +--------------------+ | |
| | | enp0s4 | | |
| | | 10.10.0.99 | | |
| | | | | |
| | | c2:67:4f:53:29:cb | | |
| +-----+--------------------+------+ |
| | +------------------+ |
| | | | |
| | | macvtap0 | |
| | | no IP | |
| +----------------------- | | |
| |c2:67:4f:53:29:cb | |
| | | |
| +------------------+ |
| | +-------------------+
| Host | | enP9p3s0 |
| | | 10.10.0.41 |
| +-------------| 00:1b:21:e3:ec:50 |
| | |
+---------------------------------------------------------------------+-------------------+
Some more details:https://github.com/cloud-hypervisor/cloud-hypervisor/discussions/5084.
We can follow the CH integration test as example to setup VM network using TAP backend. The integration test uses customized ubuntu cloudinit image. The image contains network configuration.
- echo -n "@DEFAULT_TCP_LISTENER_MESSAGE" > /dev/tcp/@HOST_IP/@TCP_LISTENER_PORT
+ echo -n "booted" > /dev/tcp/192.168.2.1/8001
The patch comes from test_infra/src/lib.rs:
prepare_cloudinit
user_data_string.replace("@DEFAULT_TCP_LISTENER_MESSAGE",DEFAULT_TCP_LISTENER_MESSAGE,);
user_data_string.replace("@TCP_LISTENER_PORT", &network.tcp_listener_port.to_string());
./scripts/create-cloud-init.sh
We can find id0 in test_data/cloud-init/ubuntu/network-config.
id0:
match:
macaddress: 12:34:56:78:90:ab
addresses:
- 192.168.2.2/24
gateway4: 192.168.2.1
There are also some other ids with different mac and ip addresses.
cp /root/workloads/focal-server-cloudimg-arm64-custom-20210929-0.raw /root/ch_files/osdisk.img
target/aarch64-unknown-linux-gnu/release/cloud-hypervisor \
--api-socket /tmp/ch0 \
--event-monitor path=/tmp/event.json \
--cpus boot=4 \
--memory size=4G,hotplug_method=virtio-mem,hotplug_size=32G \
--balloon size=0 \
--kernel /root/workloads/Image \
--disk path=/root/ch_files/osdisk.img \
--disk path=/tmp/ubuntu-cloudinit.img,iommu=on \
--net id=net123,tap=,mac=12:34:56:78:90:ab,ip=192.168.2.1,mask=255.255.255.0 \
--vsock cid=3,socket=/tmp/vsock \
--cmdline "console=hvc0 root=/dev/vda1 rw systemd.journald.forward_to_console=1"
Cloud-init detects the VM's mac address "12:34:56:78:90:ab" and sets the id0's ip address "192.168.2.2" to VM.
Issue 1:If cloudinit.img does not contain network-config, use it to start the disk for the first time, and use another cloudinit.img containing network-config for the second time to start the disk, the VM cannot automatically obtain an IP address.
Issue 2:Do not name the tap or the tap will not config the IP address "192.168.2.1" on the host, and "tap=" means that the tap is automatically named "vmtap0" on the host.
The relevant code is in test_infra/src/lib.rs:
GuestNetworkConfig
wait_vm_boot
let listener = TcpListener::bind(listen_addr.as_str()).map_err(WaitForBootError::Listen)?;
let num_events = match epoll::wait(epoll_fd, timeout * 1000_i32, &mut events[..])
listener.accept()
6. After receive "booted",I can connect the VM by ssh [email protected].
On host:
ifconfig
vmtap0: flags=67<UP,BROADCAST,RUNNING> mtu 1500
inet 192.168.2.1 netmask 255.255.255.0 broadcast 192.168.2.255
inet6 fe80::ec51:5ff:fe08:494f prefixlen 64 scopeid 0x20<link>
ether aa:16:ea:a7:03:93 txqueuelen 1000 (Ethernet)
RX packets 8 bytes 534 (534.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 16 bytes 1152 (1.1 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
On VM:
ip addr
enp0s4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 12:34:56:78:90:ab brd ff:ff:ff:ff:ff:ff
inet 192.168.2.2/24 brd 192.168.2.255 scope global enp0s4
valid_lft forever preferred_lft forever
+-------------------------------------------------------------------------------------------------------------------+
| +--------------------------------+ |
| | | |
| | VM | |
| | | |
| | +---------------------------+ | |
| | | enp0s4 192.168.2.2 | | |
| | | 12:34:56:78:90:ab | | |
| | | | | |
| +-+---------------------------+--+ |
| | |
| | |
| | |
| | |
| +---------------------------------+ |
| | | |
| | | |
| | vmtap0 | |
| | 192.168.2.1 | |
| | aa:16:ea:a7:03:93 | |
| | | |
| | | |
| +---------------------------------+ |
| |
| |
| Host |
| |
| |
| |
| |
| |
| |
+-------------------------------------------------------------------------------------------------------------------+
Some more details:https://github.com/cloud-hypervisor/cloud-hypervisor/discussions/5274 and https://cloudinit.readthedocs.io/en/latest/
On host,the IP address of enP9p3s0 is 10.10.0.9:
ip link add name virbr0 type bridge
ip link set dev virbr0 up
ip addr add 10.10.0.9/24 dev virbr0
ip route append default via 10.10.0.1 dev virbr0
ip addr flush dev enP9p3s0
ip addr flush dev vmtap0
sleep 3
ip link set dev enP9p3s0 master virbr0
ip link set dev vmtap0 master virbr0
resolvectl dns 8.8.8.8 virbr0
ip addr
2: enP9p3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master virbr0 state UP group default qlen 1000
link/ether 04:3f:72:db:60:80 brd ff:ff:ff:ff:ff:ff
92: vmtap0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr0 state UNKNOWN group default qlen 1000
link/ether 6e:df:52:f6:29:75 brd ff:ff:ff:ff:ff:ff
93: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 04:3f:72:db:60:80 brd ff:ff:ff:ff:ff:ff
inet 10.10.0.9/24 scope global virbr0
valid_lft forever preferred_lft forever
On VM:
ip addr flush enp0s4
dhclient enp0s4
ping google.com -c 4
ip addr
2: enp0s4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 12:34:56:78:90:ab brd ff:ff:ff:ff:ff:ff
inet 10.10.0.177/24 brd 10.10.0.255 scope global dynamic enp0s4
valid_lft 604631sec preferred_lft 604631sec
The VM of Ubuntu works OK with dhclient,but Fedora need more config:
ip route append default via 10.10.0.1 dev enp0s4
resolvectl dns enp0s4 8.8.8.8
resolvectl status
ping google.com -c 4
+-------------------------------------------------------------------------------------------------------------------+
| +--------------------------------+ |
| | | |
| | VM | |
| | | |
| | +---------------------------+ | |
| | | enp0s4 10.10.0.121 | | |
| | | 12:34:56:78:90:ab | | |
| | | | | |
| +-+---------------------------+--+ |
| | |
| | |
| | |
| | |
| +---------------------------------+ +--------------------------------+ |
| | | | | |
| | | | | |
| | vmtap0 | ------+| |------------------+ |
| | 192.168.2.1 | | virbr0(bridge) | | |
| | aa:16:ea:a7:03:93 | | | | |
| | | | 52:54:00:47:72:ce | | |
| | | | | | |
| +---------------------------------+ +--------------------------------+ | |
| | |
| | |
| Host +---------------------------------+
| | |
| | enP9p3s0 |
| | 10.10.0.9 |
| | |
| | 00:1b:21:e3:ec:50 |
| | |
+---------------------------------------------------------------------------------+---------------------------------+
Now you can browse the Internet.
On host:
qemu-system-aarch64 -nographic -monitor unix:/tmp/qmp-test,server,nowait \
-machine virt,gic-version=max -accel kvm -bios QEMU_EFI.fd -cpu max -smp cpus=2 -m 1G \
-drive if=none,file=/root/avocado/data/avocado-vt/images/f35-aarch64.qcow2,id=hd1 -device virtio-blk-pci,drive=hd1,bootindex=0 \
-net nic -net user
On VM:
resolvectl dns ens3 8.8.8.8
resolvectl status
ping google.com -c 4
On host
tunctl -t vmtap0
ip link add name virbr0 type bridge
ip link set dev virbr0 up
ip link set dev vmtap0 up
ip addr add 10.10.0.9/24 dev virbr0
ip route append default via 10.10.0.1 dev virbr0
ip link set dev enP9p3s0 master virbr0
ip link set dev vmtap0 master virbr0
ip addr flush dev enP9p3s0
ip addr flush dev vmtap0
resolvectl dns virbr0 8.8.8.8
qemu-system-aarch64 -nographic -monitor unix:/tmp/qmp-test,server,nowait -machine virt,gic-version=max -accel kvm \
-bios QEMU_EFI.fd -cpu max -smp cpus=2 -m 1G \
-drive if=none,file=/root/avocado/data/avocado-vt/images/f35-aarch64.qcow2,id=hd1 -device virtio-blk-pci,drive=hd1,bootindex=0 \
-net nic -net tap,ifname=tap0,script=no,downscript=no
On VM of fedora35
ping google.com
On VM of fedora38
dhclient enp0s4
resolvectl dns enp0s4 8.8.8.8
ping google.com
- Once enP9p3s0 is added to the bridge, it won't be used for routing anymore. virbr0 will take its place, so it needs an IP and have the default route attached.
- We cannot delete the IP address on enP9p3s0 before adding the interface to virbr0, otherwise network connectivity will be lost.
- However, we need to quickly remove the ip address on enP9p3s0, otherwise network connectivity will be lost after a short period.
- Linux does not allow two default routes on the same routing table. The easy workaround is just to append the new default route.
- Once the IP address of enP9p3s0 is removed, the default route attached to it is automatically removed.
- If the IP is lost automatically,you can try
systemctl stop NetworkManager && systemctl disable NetworkManager