The communication between the virtualization solutions (KVM, Xen, LXC) and the libvirt API is managed by the daemon libvirtd, which needs to run on the VM Host Server.
libvirt client applications such as virt-manager, possibly running on a remote machine, communicate with libvirtd running on the VM Host Server, which services the request using native hypervisor APIs.