cloud hypervisor - pykello/pykello.github.com GitHub Wiki

Assume we started cloud-hypervisor with --disk path=jammy.raw

  • DeviceManager::make_virtio_block_device(self, disk_cfg)
    • link to code
    • disk_cfg = {path: jammy.raw, id: None, vhost_user: false, iommu: false, direct: false, ...}
    • id is auto-assigned with _disk0
    • snapshot (result of snapshot_from_id) is None
    • image_type = block_util::ImageType::Raw

The above is called from:

  • DeviceManager::make_virtio_block_device
  • DeviceManager::make_virtio_block_devices
  • DeviceManager::make_virtio_devices
  • DeviceManager::create_devices
  • Vm::new_from_memory_manager (vm.rs:585), which calls device_manager.create_devices. Just before that, device_manager is created.

And that is called from

  • Vmm::vm_boot at vmm/src/lib.rs:636, which creates a new Vm.
  • After Vm is created, vm.boot is called.

After DeviceManager::create_devices finish, DeviceManager::add_pci_devices is called. Which calls add_virtio_pci_device, which assigns pci_segment_id (=0) and pci_device_bdf (=8). BDF means bus device function, and is formatted as BB:DD.F. So 8 means 00:01.0, and second virtio device will be at 00:02.0 which means pci_device_bdf=16.

When booting:

[    0.328035] virtio-pci 0000:00:01.0: enabling device (0000 -> 0002)
[    0.330063] virtio-pci 0000:00:02.0: enabling device (0000 -> 0002)
[    0.331437] virtio-pci 0000:00:03.0: enabling device (0000 -> 0002)
[    0.332808] virtio-pci 0000:00:04.0: enabling device (0000 -> 0002)

...

Loading, please wait...
Starting version 249.11-0ubuntu3.9
[    0.552305] cryptd: max_cpu_qlen set to 1000   
[    0.576705] virtio_blk virtio0: [vda] 4612096 512-byte logical blocks (2.36 GB/2.20 GiB)
[    0.578704] AVX2 version of gcm_enc/dec engaged.
[    0.579302] AES CTR mode by8 optimization enabled
[    0.579314]  vda: vda1 vda14 vda15
[    0.591707] virtio_blk virtio1: [vdb] 16384 512-byte logical blocks (8.39 MB/8.00 MiB)
[    0.593416]  vdb:

Rust-vmm assumes block device name is 'vda': test_run_reference_vmm.py#L366

Linux kernel in virtblk_update_capacity already knows the name of vblk (vblk->disk->disk_name): virtblk_update_capacity

Name assignment happens here: virtblk_probe Using a call to virtblk_name_format.

virtblk_probe is called by virtio_dev_probe via drv->probe.

The kernel will call virtio_dev_match to determine if a device is a virtio device, and if yes, it will call virtio_dev_probe for it.

This is registered to kernel using register_virtio_driver, which registers a bus type.

When I created an EC2 Ubuntu instance, device names where /dev/xvda, /dev/xvdb, ... These XenLinux virtual block devices and are assigned in xlvbd_alloc_gendisk. But in EC2 console it shows /dev/sda, /dev/sdb!

In create wizard I chose /dev/sdi mapping, it created /dev/xvdi mapping.

In lsblk -O -J, its subsystem is: "subsystems": "block:xen"

         "path": "/dev/xvdb",
         "maj:min": "202:16",

         "path": "/dev/xvdc",
         "maj:min": "202:32",

         "path": "/dev/xvdd",
         "maj:min": "202:48",

         "path": "/dev/xvde",
         "maj:min": "202:64",

         "path": "/dev/xvdz",
         "maj:min": "202:6400",

I Linux code:

#define XENVBD_MAJOR		202	/* Xen virtual block device */`
#define PARTS_PER_DISK		16

...
static int xen_translate_vdev(int vdevice, int *minor, unsigned int *offset)
{
...
		case XENVBD_MAJOR:
			*offset = *minor / PARTS_PER_DISK;
...
}

...

Disk Id

In Linux Kernel: virtblk_get_id, which is called by serial_show.

serial_show is exposed using DEVICE_ATTR_RO(serial).

Serial number has length restriction of 20.

#define VIRTIO_BLK_ID_BYTES	20	/* ID string length */

In qemu: https://github.com/qemu/qemu/blob/7efd65423ab22e6f5890ca08ae40c84d6660242f/hw/block/virtio-blk.c#L1057 In spdk: https://github.com/spdk/spdk/blob/1334e3e40700b9f9f9c9d357bc748a33fb08797a/lib/vhost/vhost_blk.c#L625 In cloud-hypervisor:

Stacktrace:

  • block_util::build_disk_image_id
  • virtio_devices::block::{impl#6}::activate
  • virtio_devices::transport::pci_device::VirtioPciDeviceActivator::activate
  • vmm::vm::Vm::activate_virtio_devices