KVM video card transparent transmission -- the road of building a dream

precondition

Graphics card pass through relies on some virtualization technologies , May not be available on your machine , The following conditions must be checked first :

• First CPU Must support hardware virtualization (for KVM) and IOMMU(for Straight through )

• • Compatible Intel CPUs list (Intel VT-x and Intel VT-d). It can be understood as Intel VT-x yes CPU Virtualization of ,VT-d yes I/O Virtualization of devices , The two things are different .

  • Bulldozer And higher （ Include Zen） All of the AMD CPU Should be compatible .

• The motherboard must support IOMMU

• • Chipsets and BIOS Must all support IOMMU. It is difficult to see directly whether it supports , But there is a relatively complete list for query ：Xen Wiki  and  Wikipedia:List of IOMMU-supporting hardware

• GPU ROM Must support UEFI.2012 Years later GPU Almost all support .

A more convenient way to check is to BIOS Find and open... In settings VT-d (Intel) perhaps IOMMU (AMD). Without these options , That is not to support .

stay KVM Start on host IOMMU

#  Edit file first 
$ vim /etc/default/grub 
#  The first 6 That's ok :  add to  intel_iommu=on ( If it is AMD CPU,  add to  [amd_iommu=on])
#  You should also add parameters  iommu=pt,  prevent  Linux  Contact with equipment that cannot be transmitted through 
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet intel_iommu=on iommu=pt"
GRUB_DISABLE_RECOVERY="true"

$ grub2-mkconfig -o /boot/grub2/grub.cfg
$ reboot  #  restart 

Verify after restart IOMMU Is it enabled? ：

$ dmesg | grep -E "DMAR|IOMMU"
[    0.000000] ACPI: DMAR 00000000bf79e0d0 00118 (v01    AMI  OEMDMAR 00000001 MSFT 00000097)
[    0.000000] DMAR: IOMMU enabled
[    0.069846] DMAR: Host address width 40
[    0.069848] DMAR: DRHD base: 0x000000fbffe000 flags: 0x1
[    0.069856] DMAR: dmar0: reg_base_addr fbffe000 ver 1:0 cap c90780106f0462 ecap f020fe
[    0.069858] DMAR: RMRR base: 0x000000000ec000 end: 0x000000000effff
[    0.069859] DMAR: RMRR base: 0x000000bf7ec000 end: 0x000000bf7fffff
[    0.069861] DMAR: ATSR flags: 0x0
[    0.069863] DMAR-IR: IOAPIC id 6 under DRHD base  0xfbffe000 IOMMU 0
[    0.070128] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.960840] DMAR: dmar0: Using Queued invalidation
[    0.960860] DMAR: Setting RMRR:
[    0.960890] DMAR: Setting identity map for device 0000:00:1a.0 [0xbf7ec000 - 0xbf7fffff]
[    0.960925] DMAR: Setting identity map for device 0000:00:1a.1 [0xbf7ec000 - 0xbf7fffff]
[    0.960958] DMAR: Setting identity map for device 0000:00:1a.2 [0xbf7ec000 - 0xbf7fffff]
[    0.960994] DMAR: Setting identity map for device 0000:00:1a.7 [0xbf7ec000 - 0xbf7fffff]
[    0.961031] DMAR: Setting identity map for device 0000:00:1d.0 [0xbf7ec000 - 0xbf7fffff]
[    0.961064] DMAR: Setting identity map for device 0000:00:1d.1 [0xbf7ec000 - 0xbf7fffff]
[    0.961100] DMAR: Setting identity map for device 0000:00:1d.2 [0xbf7ec000 - 0xbf7fffff]
[    0.961132] DMAR: Setting identity map for device 0000:00:1d.7 [0xbf7ec000 - 0xbf7fffff]
[    0.961153] DMAR: Setting identity map for device 0000:00:1a.0 [0xec000 - 0xeffff]
[    0.961167] DMAR: Setting identity map for device 0000:00:1a.1 [0xec000 - 0xeffff]
[    0.961182] DMAR: Setting identity map for device 0000:00:1a.2 [0xec000 - 0xeffff]
[    0.961196] DMAR: Setting identity map for device 0000:00:1a.7 [0xec000 - 0xeffff]
[    0.961210] DMAR: Setting identity map for device 0000:00:1d.0 [0xec000 - 0xeffff]
[    0.961224] DMAR: Setting identity map for device 0000:00:1d.1 [0xec000 - 0xeffff]
[    0.961238] DMAR: Setting identity map for device 0000:00:1d.2 [0xec000 - 0xeffff]
[    0.961255] DMAR: Setting identity map for device 0000:00:1d.7 [0xec000 - 0xeffff]
[    0.961270] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.961287] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.961424] DMAR: Intel(R) Virtualization Technology for Directed I/O

  start-up  vfio-pci  The kernel module

Retain RHEL/CentOS The general kernel provided by the official is OK

$ uname -a

#  Display the... Of the graphics card PCI Certification numbers and [ supplier ID: equipment ID]
$ lspci -nn | grep -i nvidia
3b:00.0 3D controller [0302]: NVIDIA Corporation GP104GL [Tesla P4] [10de:1bb3] (rev a1)
86:00.0 3D controller [0302]: NVIDIA Corporation GP104GL [Tesla P4] [10de:1bb3] (rev a1)

#  edit vfio The configuration file 
$ vim /etc/modprobe.d/vfio.conf
#  Create a new line , Appoint ids= supplier ID: equipment ID
options vfio-pci ids=10de:1bb3,10de:1bb3

#  Create a new file , write in  vfio-pci
$ echo 'vfio-pci' > /etc/modules-load.d/vfio-pci.conf

$ reboot  #  restart 

Check after restart vfio Whether to load

$ dmesg | grep -i vfio
[    6.441900] VFIO - User Level meta-driver version: 0.3
[    6.497125] vfio-pci: probe of 0000:3b:00.0 failed with error -22
[    6.497163] vfio-pci: probe of 0000:86:00.0 failed with error -22
[    6.497181] vfio_pci: add [10de:1bb3[ffff:ffff]] class 0x000000/00000000
[    6.497233] vfio-pci: probe of 0000:3b:00.0 failed with error -22
[    6.497249] vfio-pci: probe of 0000:86:00.0 failed with error -22
[    6.497258] vfio_pci: add [10de:1bb3[ffff:ffff]] class 0x000000/00000000

to update QEMU

#  see QEMU edition 
$ /usr/libexec/qemu-kvm -version
QEMU emulator version 1.5.3 (qemu-kvm-1.5.3-175.el7_9.3), Copyright (c) 2003-2008 Fabrice Bellard

#  Upgrade new version 
$ yum -y install centos-release-qemu-ev
Running transaction
  Installing : centos-release-virt-common-1-1.el7.centos.noarch                                                    1/2
  Installing : centos-release-qemu-ev-1.0-4.el7.centos.noarch                                                      2/2
  Verifying  : centos-release-qemu-ev-1.0-4.el7.centos.noarch                                                      1/2
  Verifying  : centos-release-virt-common-1-1.el7.centos.noarch                                                    2/2

Installed:
  centos-release-qemu-ev.noarch 0:1.0-4.el7.centos

Dependency Installed:
  centos-release-virt-common.noarch 0:1-1.el7.centos
  
#  close  CentOS-QEMU-EV
$ sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-QEMU-EV.repo

#  take  qemu-kvm  Replace with  qemu-kvm-ev 
$ yum --enablerepo=centos-qemu-ev -y install qemu-kvm-ev
$ systemctl restart libvirtd

#  see  KVM  edition 
$ /usr/libexec/qemu-kvm -version
QEMU emulator version 2.12.0 (qemu-kvm-ev-2.12.0-44.1.el7_8.1)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

#  Verify that  q35
$ /usr/libexec/qemu-kvm -machine help | grep q35
q35                  RHEL-7.6.0 PC (Q35 + ICH9, 2009) (alias of pc-q35-rhel7.6.0)
pc-q35-rhel7.6.0     RHEL-7.6.0 PC (Q35 + ICH9, 2009)
pc-q35-rhel7.5.0     RHEL-7.5.0 PC (Q35 + ICH9, 2009)
pc-q35-rhel7.4.0     RHEL-7.4.0 PC (Q35 + ICH9, 2009)
pc-q35-rhel7.3.0     RHEL-7.3.0 PC (Q35 + ICH9, 2009)

Create a new virtual machine

$ virt-install \
--name centos7 \
--ram 8192 \
--disk path=ubuntu16.img,size=30 \
--vcpus 4 \
--os-type linux \
--os-variant ubuntu16.04 \
--network bridge=br0 \
--graphics none \
--console pty,target_type=serial \
--location ubuntu-16.04.7-server-amd64.iso \
--extra-args 'console=ttyS0,115200n8 serial' \
--host-device 3b:00.0 \
--features kvm_hidden=on \
--machine q35

 Parameter interpretation ：

name  Virtual machine name 
ram  Memory size 
disk  Specify the storage device and type 
vcpus  fictitious  CPU  Number 
os-type  Type of operating system 
os-variant  Specific operating system variants 
network  The bridge or network connecting the host computer 
graphics  Virtual machine display provider , The default is  vnc
console ??
location  Install source ,  Support  FTP、HTTP  And  NFS  etc. 
extra-args  When execution from  –location  Option to specify the location of the client installation , Attach kernel command line parameters to the installer 
host-device  Attach physical devices to clients 
features kvm_hidden=on  Hide the virtual machine state from the host 
machine QEMU Simulated CPU framework