IOMMU (VII) -vfio and mdev

Have a look first vifo and mdev What modules does the kernel have , Then analyze the relationship between modules .

vfio It is an interface between user mode and kernel mode , Be similar to vhost and vhost-user, The specific usage is vfio.ko adopt /dev/vfio In the kernel, the user mode is provided with api,qemu perhaps dpdk use api Interact with kernel modules .

vfio-pci Is a kernel driver , Network card and NVME Disk and other devices can use this driver , Use vfio-pci Will be called to vfio-pci Of probe.vfio-pci For standard pci equipment , If multiple virtual machines want to use this pci The device is turned on pci The equipment sr-iov function , This pci The device becomes multiple standard pci equipment , One for each virtual machine .sr-iov It is a segmentation method of hardware resources , Not only DMA queue Such data slicing ,pci config space And so on , The hardware implementation is too complex ,GPU and NVME It is difficult for these devices to realize the segmentation of the control surface , But there are other ways to segment the data plane , Different virtual machines can share data plane functions and ensure isolation , Call this device mdev equipment , It's strict mediated Devices that can be virtualized .mdev.ko Provide users with interfaces to create virtual mdev equipment , Make virtual mdev And reality mdev relation , Put the real mdev It is called the parent device , At the same time, the virtual mdev And drive vfio-mdev matching ,vfio-mdev Begin probe.

vfio-pci and mdev-vfio Of probe All call vfio_add_group_dev Add your own ops, Generate a dev,qemu adopt /dev/vfio Access to this dev, Operate this again dev Is called to vfio-pci and vfio-mdev Of ops function ,vfio-pci work ,vfio-mdev Basically nothing , Call the parent device as soon as you change hands ops work , It is equivalent to calling GPU perhaps NVME The driver .sr-iov yes pci Standard device virtualization solutions ,mdev For the device virtualization solution privately owned by the manufacturer , The driver is implemented by the manufacturer , Hardware is also made by manufacturers , The combination of driver and hardware can achieve the effect of virtualization .

[[email protected] 81]# ll /sys/class/vfio/81/dev
-r--r--r--. 1 root root 4096 May 28 10:31 /sys/class/vfio/81/dev

vfio group Not a concept created out of thin air ,vfio group and IOMMU The hardware group Closely related , therefore vfio Another important function is vfio_register_iommu_driver,vfio_iommu_type1.ko Just call this function to vfio Action registered IOMMU Of ops, One device DMA Use of pagetable You can go through this ops Configuration to IOMMU in .

qemu call memory_region_init_io register vfio-pci The equipment config space, Drive write in virtual machine config space,qemu Intercept and simulate .

pci_host_data_write->pci_data_write->pci_host_config_write_common->vfio_pci_write_config->vfio_msi_enable

qemu Call kernel vfio,irqbypass Used for holding vfio and kvm Connect .

vfio_msi_enable->vfio_enable_vectors(qemu Code )->vfio_pci_set_irqs_ioctl( kernel vfio Code )->vfio_pci_set_msi_trigger->vfio_msi_set_block->vfio_msi_set_vector_signal->irq_bypass_register_producer->pi_update_irte

qemu Call to the kernel at the same time kvm.

vfio_msi_enable->vfio_add_kvm_msi_virq->kvm_irqchip_add_irqfd_notifier_gsi->kvm_irqchip_assign_irqfd(qemu Code )->kvm_vm_ioctl(kvm Code )->kvm_irqfd->kvm_irqfd_assign->irq_bypass_register_consumer

The peripheral interrupt is coming ,vfio Handle ,signal kvm,kvm Then inject the virtual interrupt into the virtual machine .

vfio_msihandler->eventfd_signal wakeup 了 irqfd_wakeup->schedule_work

irqfd_inject->kvm_set_irq Interrupt to register