Virtual machine online migration based on openstack

Virtual machine migration refers to the movement of virtual machine operating systems and applications running on the source physical host to the target physical host or virtual machine , And it can run normally on the target host . stay openstack in ,openstack The migration function of the built-in virtual machine , Allow one to be running Virtual machine instance from a compute node Move to another one compute node. Now I would like to recommend a relevant practical case , The author is the technical director of UFIDA public cloud , Senior member of China computer society , Member of the big data special committee , Former associate researcher of the Chinese Academy of Sciences .

The following is the author's original text ：

I have always wanted to share with you the online migration of virtual machines , Considering the stability , After we have been running online for several months, it is stable , Then sum it up and share with you .

Describe the scenario roughly ： The system adopts a computing storage loosely coupled structure , The image file of the virtual machine is on the remote shared storage , So the migration speed is very fast . In our system , The fastest one used 6 second , That is, the online migration has been completed , This is real live migration, As we move , While deliberately writing data in the virtual machine , Also completed normally .

Configuration scheme

1. modify Nova.conf file

add to ： image_cache_manager_interval=0 live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_UNSAFE modify ： vncserver_listen=0.0.0.0

2. The names of the participating computing node machines can ping through .

3. Modify the calculation node /etc/libvirt/libvirtd.conf: 

before : #listen_tls = 0 after : listen_tls = 0 before : #listen_tcp = 1 after : listen_tcp = 1 add: auth_tcp = "none" 

4. modify /etc/sysconfig/libvirtd:

before :# LIBVIRTD_ARGS="--listen" after :LIBVIRTD_ARGS="–listen"

5. Modify the virtual machine to be migrated on the source computing node /var/run/libvirt/qemu/instance–xxx.xml file , Delete migrate-qemu-fd This business , take vnc Change the parameter to 0.0.0.0

6. Restart the compute node nova

7 remarks ：

1. Because the cloud machine has not been configured for online migration , Before migrating the virtual machine , The virtual machine needs to be restarted . 2.. Because on the computing node libvirtd In the configuration of auth_tcp="none", It's a security vulnerability , We need to find a safer way , Or after the migration , Comment out the line , restart libvirt 3 An auxiliary program has been written for automatic migration

Problems encountered and solutions

1. Virtual disk Of cache mode by writethrough, An error is reported during migration

openstack Think in CentOS On disk cache mod by writethrough when , Migration is not secure . The solution ： stay nova.conf live_migration_flag Add... After the parameter VIR_MIGRATE_UNSAFE, There is no such parameter in the official online migration configuration file .

2.qemu1.4 One of the bug Causes the migration to fail

Migration failed , On the destination node /var/log/libvrit/qemu/instances--xxxx.log in ： char device redirected to /dev/pts/1 (label charserial1) qemu: warning: error while loading state section id 2 load of migration failed terms of settlement ： 1. Modify the virtual machine to be migrated on the source computing node /var/run/libvirt/qemu/instance–xxx.xml Deleted in the text migrate-qemu-fd This business 2. Restart the on the source compute node libvirtd 3. And then execute nova live-migration command This operation has been programmed to execute automatically .

3.vncserver The problem of , You need to restart the virtual machine before you can migrate .

Because before Nova.conf in vncserver_listen= Computer node ip, So in the virtual machine Kvm In process parameter vnc= Compute the ip, An error is reported during migration , The destination node cannot be bound to the source node IP, So it needs to be modified Libvirt.xml The configuration file , Restart the virtual machine , Then you can migrate . terms of settlement ： 1. On the source compute node /var/run/libvirt/qemu/instance–xxx.xml In the text vnc The parameter of is modified to 0.0.0.0 2. Restart the source compute node libvirtd 3. And then execute nova live-migration command This operation has been programmed to complete automatically

4. After migration console.log,disk The owner became root/root

After migration , Find the virtual machine console.log and disk The document is owned by qemu.qumu Turned into root.root, This estimate is openstack Problems with the migration program , This problem does not affect the virtual machine at present . terms of settlement : The owner of the modified document , This operation has been programmed to complete automatically

5. Source and destination nodes cpu Incompatibilities

Migration failed , stay /var/log/nova/compute Log ： "InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.\n\n0\n\nRefer to <a href="http://libvirt.org/html/libvirt-libvirt.html#virCPUCompareResult\n" ]"="" style="cursor: pointer; color: rgb(0, 102, 204); text-decoration: none;">http://libvirt.org/html/libvirt-libvirt.html#virCPUCompareResult\n"] terms of settlement ： There is no solution yet

6. The destination node memory is negative ,

Migration failed , From the control node api.log The error output on is that the memory of the destination node is -3400, Can not meet the needs of migration . terms of settlement ： use nova The command specifies to create a virtual machine on the computing node , Can succeed . It is estimated that the scheduling algorithm at the time of migration is inconsistent with that at the time of virtual machine creation .

7. Error log , stay 2.4 On api.log

The logs that you usually see during migration are ： 1. On the destination node /var/log/libvirt/qemu/instances–xxx.log 2. On the destination node /var/log/nova/compute.log 3. On the source node /var/log/nova/compute.log Sometimes migration fails , An error is reported after the command line is executed ： ERROR: Live migration of instance bd785968-72f6-4f70-a066-b22b63821c3b to host compute-13 failed (HTTP 400) (Request-ID: req-180d27b5-9dc7-484f-9d9a-f34cccd6daa2) However, no error messages can be seen in the above three log files . terms of settlement ： On the control node or the node that operates the migration command /var/log/nova/api.log There is an error message

The detour

1. Try not to modify nova.conf Inside vncserver_listen Parameter is 0.0.0.0,

take /var/run/log/libvirt/qemu/instances--xxx.log Inside vnc Change to the destination node ip, restart libvritd, Then move on , Can succeed , But if the migration fails , When you need to restart the virtual machine , Virtual machine startup failed , stay /var/log/libvrit/qemu/instances-xx.log The mistake is Failed to start VNC server on `172.18.2.15:0': Failed to bind socket: Cannot assign requested address and /mnt/instances/instance--xxx/libvirt.xml There is no change to the destination node in Ip. I don't know where this parameter is saved

2.vnc Port problem

After a migration failure , At the destination node /var/log/libvirt/qemu/instance--xxx.log The mistake in is ： 2013-11-05 05:42:39.401+0000: shutting down qemu: terminating on signal 15 from pid 10271, The guess is that the virtual machine is on the source node vnc The listening port is occupied on the destination node , So it can not be started . Later, it was tested on other machines and found that , After migrating to the destination node vnc The port of can be adjusted by itself .

Reference resources ：

How to Migrate an Instance with Zero Downtime: OpenStack Live Migration with KVM Hypervisor and NFS Shared Storage