Criu enables hot migration

Introduce CRIU

CRIU（Checkpoint/Restore In Userspace） yes Linux A software on . It can be used to pause a running container or process , according to CRIU Pause the generated file and resume the container or process from the breakpoint , And then go ahead and do it .

Environmental installation

The experiment involved two computers （ virtual machine ） So the communication between the two computers uses NFS（network file system）.

Container on use docker 17.06.0.

CRIU Version selection for 3.14.

Operating system selection of two virtual machines Ubuntu16, tried Ubuntu20 It doesn't feel right .

install NFS

The server IP：192.168.40.132

client IP：192.168.40.134

First, explain the server-side installation nfs The process of . Use apt-get Download complete nfs-kernel-server after , Add the directory to be mounted to the client to /etc/exports In file .

# config host network
sudo apt-get install nfs-kernel-server -y
# config NFS
sudo gedit /etc/exports
# Add the below line to exports file:
# /home 192.168.40.134(rw,sync,no_root_squash,no_subtree_check)
# Then restart nfs-kernel-server service
sudo systemctl restart nfs-kernel-server

As needed during the experiment , On the server exports The document added /home/proc and /home/container Catalog , Store process migration separately 、 Files related to container migration .

After completing the above configuration , Configure the firewall . The client should be allowed （IP by 192.168.40.134） The interview of .

# config firewall
# First, check firewall status
sudo ufw status
# If ufw is inactive, use the below command to enable ufw:
sudo ufw enable
# Make ufw allow incoming and outgoing:
sudo ufw default allow incoming
sudo ufw default allow outgoing
# Make client server can access host server
sudo ufw allow from 192.168.40.134 to any port nfs

The code for configuring the client is as follows . Only... Is attached /home Catalog , Other directory mounts can be adjusted as needed .

#!/bin/bash
# install nfs

sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak
sudo rm -f /etc/apt/sources.list
sudo cp ./sources.txt /etc/apt/sources.list
echo "finish changing source"
sudo apt update
sudo apt upgrade -y -f

# config host network
sudo apt-get install nfs-common -y
# mount the host /home
sudo mkdir -p /nfs/home
sudo mount 192.168.40.132:/home /nfs/home
# check mount status
sudo df -h

install Docker

docker Also use apt install , Not source code compilation . First of all, will docker（ and CRIU） All required supporting software are installed , The code is as follows ：

#build dependent
sudo apt install build-essential -f -y
sudo apt install pkg-config -f -y
sudo apt install libnet-dev python-yaml libaio-dev -f -y
sudo apt install libprotobuf-dev libprotobuf-c-dev protobuf-c-compiler protobuf-compiler python-protobuf libnl-3-dev libcap-dev libbsd-dev -f -y
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common -y -f

After the docker Join in sources.list, And download the corresponding version of docker. Download here docker Version is 17.06.0, Download the old version of docker It's to make sure that checkpoint Does not appear bug（ Use apt-cache madison docker-ce Command to view all downloadable versions ）. Finally, if the download succeeds hello-world And the operation indicates that the installation is successful .

# install docker
# remove docker
sudo apt-get remove docker docker-engine docker.io containerd runc 
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
# add repo
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable"
# update and install 
sudo apt-get update
sudo apt-get install docker-ce=17.06.0~ce-0~ubuntu -y
# check 
sudo docker run hello-world

installation is complete docker Then you need to turn on experiment attribute , The code is as follows ：

# add experiment to docker
sudo echo "{\"experimental\": true}" >> /etc/docker/daemon.json
sudo systemctl restart docker

install CRIU

CRIU The installation uses the source code compilation installation method . Download the CRIU The version is the latest 3.14 edition . Compiling the source code may require the installation of other dependent software , You can add... As prompted . Finally, if criu check The output is Looks good. Explain that there is no problem .

# criu install
curl -O -sSL http://download.openvz.org/criu/criu-3.14.tar.bz2
tar xjf criu-3.14.tar.bz2
cd criu-3.14
make
sudo cp ./criu/criu /usr/local/bin
# check
sudo criu check

Process hot migration

Process hot migration starts by creating a process to be migrated on the server test.h（ The process keeps outputting timestamps ） because CRIU The process of process hot migration cannot depend on the current shell window , So you need to use setsid Instructions will be time.sh Go out on your own , Redirect both output and input . The instructions are as follows . After running this command, you can see test.out Time stamps are constantly written in , When you can tail Command view .

$ setsid ./test.sh </dev/null &>$PWD/test.out

Script test.sh The contents are as follows ：

#! /bin/bash

while (true)
do
	time=$(date +%s.%N)
    echo $time
	sleep 10e-4
done

Pause the process on the server to determine the pid, Therefore use ps aux View the progress of pid. Then use CRIU Of dump Execute the pause process .

id=$(ps -ef | grep time| grep -v grep | awk '{print $2}')
criu dump --tree $id --images-dir $PWD

After the server pauses the process , Reply to the process in the client , Because the directory has not changed , So directly specify –images-dir Is the current directory （$PWD） that will do .

criu restore --images-dir $PWD

CRIU The general idea of the heat transfer process is ： Pause the process on the server , Then use... On the client CRIU Recover outgoing process . Use tail Command view test.out Files will be found using CRIU After pausing the process test.out No new input , After recovery on the client, it will go to test.out Enter the timestamp in . The whole process takes place on the server /home/proc Directory , This directory will be attached to the client /home/proc Directory . Note that you must create the same file path on both virtual machines , Otherwise, it will not succeed

Container heat transfer

Container thermal migration is similar to process thermal migration , Just add docker The operation of . The basic idea is to create containers on the server , Then pause the container , View the output of the run . View the output after the client recovers the container , If the running output is immediately followed by the server output, the hot migration is successful . The whole process takes place on the server /home/container Under the table of contents , The client will the server's /home/container The directory is mounted to the local /home/container Next .

use first docker Create a container looper2, The program incrementally outputs i, Output once every second .

docker run -d --name looper2 --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'

Interrupt the program after waiting for output for a period of time , The corresponding directory will be generated after the program is interrupted , The operating state of the storage container , Use when restoring containers .

sudo docker checkpoint create --checkpoint-dir=/home/container/ looper2 checkpoint2

see logs as follows ：

sudo docker logs looper2

Use... On the client docker Restore the container , Note that there –checkpoint-dir Just specify to checkpoints directory , No further development is required checkpoint2（ Already by –checkpoint Appoint ）

docker create --name looper-clone --security-opt seccomp:unconfined busybox /bin/sh -c 'i=0; while true; do echo $i; i=$(expr $i + 1); sleep 1; done'
docker start --checkpoint-dir=/home/container/1101bde4f985176cab26a3fae2957763c35be3c8d0c27f0fa9e469c1a4ff5928/checkpoints/ --checkpoint=checkpoint2 looper-clone

After replying to the container, you can use the same command to view the running results . as follows ：