Learning about Android Virtualization Framework - liuyq/android-issues GitHub Wiki

  1. Enabling KVM kernel configs

    CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQFD=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_MMIO=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_HAVE_KVM_ARCH_TLB_FLUSH_ALL=y
CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
CONFIG_HAVE_KVM_IRQ_BYPASS=y
CONFIG_HAVE_KVM_VCPU_RUN_PID_CHANGE=y
CONFIG_KVM_XFER_TO_GUEST_WORK=y
CONFIG_KVM=y
CONFIG_PTP_1588_CLOCK_KVM=y

    but only these configs enabled do not mean that the kvm mode is available when the board booted up. hardware features and bootloader should support it as well

    I kvm [1] : HYP mode not available

    which is checked in arch/arm64/kvm/arm.c kvm_arch_init

    if (!is_hyp_mode_available()) {
    kvm_info("HYP mode not available\n");
    return -ENODEV;
}

    while booted up successfully device with the /dev/kvm

    I kvm [1] : IPA Size Limit: 40 bits
I kvm [1] : vgic interrupt IRQ9
I kvm [1] : Hyp mode initialized successfully

  2. Crosvm KvmKernelIrqChip source

    external/crosvm/devices/src/irqchip/kvm/aarch64.rs if you want to understand the KvmKernelIrqChip related problems

    # drop kvm-arm.mode=protected would help for this problem
E crosvm  : exiting with error 1: failed to create IRQ chip
E kvm [1] : GICv2 not supported in protected mode

  3. Data abort outside memslots with no valid syndrome info

    when run crosvm with the u-boot bootloader

    $ adb shell //data/local/tmp/trs/android-crosvm.sh
+ d_vm=trs 
+ /apex/com.android.virt/bin/crosvm --extended-status --log-level 'debug,disk=off' run --disable-sandbox --no-balloon --cpus 1 '--serial=type=stdout,hardware=serial,num=1,stdin,console' '--serial=type=stdout,hardware=serial,num=2' '--serial=type=stdout,hardware=virtio-console,num=1' '--serial=type=stdout,hardware=virtio-console,num=2' '--serial=type=stdout,hardware=virtio-console,num=3' --disk /data/local/tmp/trs/trs-image-trs-qemuarm64.wic --bios /data/local/tmp/trs/u-boot.bin
[2023-01-20T12:46:35.442404950+00:00 INFO  crosvm] crosvm started.
[2023-01-20T12:46:35.445452345+00:00 INFO  crosvm] CLI arguments parsed.
[2023-01-20T12:46:35.446109637+00:00 DEBUG crosvm::crosvm::sys::unix] creating Kvm hypervisor
[2023-01-20T12:46:35.456742450+00:00 INFO  crosvm::crosvm::sys::unix::device_helpers] Trying to attach block device: /data/local/tmp/trs/trs-image-trs-qemuarm64.wic
[2023-01-20T12:46:35.643517970+00:00 ERROR crosvm::crosvm::sys::unix::vcpu] vcpu hit unknown error: Function not implemented (os error 38)
[2023-01-20T12:46:35.644364325+00:00 INFO  crosvm::crosvm::sys::unix] vcpu crashed
[2023-01-20T12:46:35.644518491+00:00 ERROR crosvm::crosvm::sys::unix::vcpu] failed to send VcpuControl: sending on a closed channel
[2023-01-20T12:46:35.645625262+00:00 ERROR devices::utils::event_loop] removing event handler due to error: host backend device provider failed: failed to read control tube: tube was disconnected
[2023-01-20T12:46:35.688130991+00:00 INFO  crosvm] exiting with crash
$

    The following error will be reported on the serial console side:

    # from arch/arm64/kvm/mmio.c io_mem_abort
# [ 1731.013283][ T2460] kvm [2457]: Data abort outside memslots with no valid syndrome info

# source code
int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
{
     ...
     /*
      * No valid syndrome? Ask userspace for help if it has
      * volunteered to do so, and bail out otherwise.
      *
      * In the protected VM case, there isn't much userspace can do
      * though, so directly deliver an exception to the guest.
      */
     if (!kvm_vcpu_dabt_isvalid(vcpu)) {
         if (is_protected_kvm_enabled() &&
             kvm_vm_is_protected(vcpu->kvm)) {
             kvm_inject_dabt(vcpu, kvm_vcpu_get_hfar(vcpu));
             return 1;
         }

         if (test_bit(KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER,
                  &vcpu->kvm->arch.flags)) {
             run->exit_reason = KVM_EXIT_ARM_NISV;
             run->arm_nisv.esr_iss = kvm_vcpu_dabt_iss_nisv_sanitized(vcpu);
             run->arm_nisv.fault_ipa = fault_ipa;
             return 0;
         }

         kvm_pr_unimpl("Data abort outside memslots with no valid syndrome info\n");
         return -ENOSYS;
     }
     ...
}

    here are some u-boot files might be related:

    board/emulation/common/Kconfig: CROSVM_MEM_MAP <====== especially this one
board/emulation/qemu-arm/Kconfig: SYS_EARLY_PCI_INIT
board/emulation/qemu-arm/qemu-arm.c

boot/Kconifg: FORCE_SECURE_BOOT
boot/Makefile: CONFIG_ANDROID_BOOTLOADER CONFIG_ANDROID_BOOTLOADER_KEYMINT_CONSOLE

cmd/Makefile: CONFIG_CMD_BOOT_ANDROID

common/Kconfig: ANDROID_BOOTLOADER_KEYMINT_CONSOLE/ANDROID_BCC
drivers/firmware/Kconfig: KVM_HYP_SERVICES
drivers/virtio/Kconfig: VIRTIO_CONSOLE

include/configs/qemu-arm.h
include/linux/arm-smccc.h
include/android_image.h

  4. how the command line for crosvm is generated:

    packages/modules/Virtualization/virtualizationservice/src/crosvm.rs, help understand the meaning of the options, like the --no-balloon options and the serials.

    ["--extended-status", 
     "--log-level", "info,disk=off", 
     "run", 
     "--disable-sandbox", 
     "--cid", "2051", 
     "--no-balloon", 
     "--cpus", "1",
     # /dev/ttyS0 uart device: used as the output device by bootloaders and as early console by linux 
     "--serial=type=file,path=/proc/self/fd/17 (/data/local/tmp/qemu/console.log),hardware=serial,num=1",
     # /dev/ttyS1 uart device: used to report the reason for the VM failing.
     "--serial=type=file,path=/proc/self/fd/8,hardware=serial,num=2",
     # /dev/hvc0 virtio-console device: used as the console device where kmsg is redirected to
     "--serial=type=file,path=/proc/self/fd/17 (/data/local/tmp/qemu/console.log),hardware=virtio-console,num=1",
     # /dev/hvc1 virtio-console device: used as the ramdump output
     "--serial=type=file,path=/proc/self/fd/21 (/data/misc/virtualizationservice/2051/ramdump),hardware=virtio-console,num=2",
     # /dev/hvc2 virtio-console device: used as the logcat output
     "--serial=type=file,path=/proc/self/fd/18 (/data/local/tmp/qemu/vm.log),hardware=virtio-console,num=3",
     "--initrd", "/proc/self/fd/23 (/data/local/tmp/qemu/rootfs.ext4)", 
     "--params", "root=/dev/vda", 
     "/proc/self/fd/22 (/data/local/tmp/qemu/Image)", 
     "--socket", 
     "/proc/self/fd/9", 
     "--params", 
     "crashkernel=17M"]

  5. how the composite image will be created by AVF:

    packages/modules/Virtualization/virtualizationservice/src/aidl.rs, when different partitions are specified with different image files.

  6. help output of the crosvm command

    external/crosvm/src/crosvm/cmdline.rs, like descriptions for for the SerialParameters

  7. VmConfig defined in packages/modules/Virtualization/libs/vmconfig/src/lib.rs

    where you could check all the possible configurations and the configs validation

  8. some other links:

    crosVM分析, crosVM分析 Link CSDN

    virtio iommu, virtio iommu CSDN

    KVM(Kernel-based Virtual Machine)源码分析 KVM 虚拟化原理2— QEMU启动过程