Explain kubernetes backup and recovery tools velero | learn more about carina series phase III

Carina  It is a cloud native local storage project led and initiated by Boyun （GitHub The address is ： https://github.com/carina-io/carina）, At present, it has entered CNCF Panorama .
Carina  It can provide services for stateful applications in the cloud native environment High performance 、 No operation and maintenance Of Local storage solutions , With storage volume lifecycle management 、LVM/RAW Panel supply 、 Intelligent scheduling 、RAID management 、 Automatic layering and other capabilities , Designed to provide very low latency for cloud native stateful Services 、 No operation and maintenance 、 Understand database data storage system .Carina As one of the components of Bo cloud container cloud platform , It has operated stably in the production environment of multiple financial institutions for many years .

There are two traditional data backup schemes , One is to use the server storing data to realize snapshot based backup , The other is to deploy a proprietary backup on each target server agent And specify the backup data directory , Periodically copy data to external storage . The backup mechanisms of these two methods are relatively fixed , In the age of cloud primordial, it cannot adapt to the elasticity after containerization 、 Deployment scenarios such as pooling .

Cloud native storage plug-ins Carina For example , In data sensitive scenarios such as databases, each database cluster includes multiple computing instances , The instance may drift arbitrarily in the cluster and realize automatic fault recovery . The traditional data backup method can rapidly expand and shrink the capacity of the database cluster 、 In scenarios such as cross node drift, it is unable to automatically follow the migration of computing instances, resulting in data backup failure , So a fit k8s Backup tools for container scenarios are very important .

Kubernetes Backup and recovery tools ：velero

Velero It is a disaster recovery and migration tool in the cloud native era , use Go Language writing , And in github Open source on , The open source address is ：https://github.com/vmware-tanzu/velero.Velero From Spanish , It means sailing boat , Very much in Kubernetes The naming style of the community .

utilize velero Users can back up safely 、 Recovery and migration Kubernetes Cluster resources and persistent volumes . Its basic principle is to cluster data , For example, cluster resources and persistent data volumes are backed up to the object store , Pull data from the object store during recovery . In addition to disaster recovery, it can also do resource transfer , Support the migration of container applications from one cluster to another , This is also velero A very successful use case .

Velero It mainly includes two core components , They are server and client respectively . The server runs on a specific Kubernetes In the cluster , The client is a command line tool that runs locally , As long as the configuration is good kubectl And kubeconfig You can use , It's simple .

Velero Based on its implementation kubernetes Resource backup capability , Can be easily implemented Kubernetes Data backup and recovery of cluster 、 Copy kubernetes Cluster resources to other kubernetes Cluster or quickly copy the production environment to the test environment .

In terms of resource backup ,velero Support data backup to numerous cloud storage , for example AWS S3 or S3 Compatible storage systems 、Azure Blob、Google Cloud Storage 、Aliyun OSS etc. . And back up the entire kubernetes The data storage engine of etcd comparison ,velero The control of is more detailed , It can be done to Kubernetes Backup at the object level within the cluster , It can also be done by Type、Namespace、Label Such as object classification backup or recovery .

Velero Workflow

Take the core data backup as an example , When executed velero backup create my-backup when ：

• Velero The client first calls Kubernetes API Server to create Backup object ;
• BackupController Will receive notification of new Backup Objects are created and validated ;
• BackupController Start the backup process , It's through a query API Server to obtain resources to collect data for backup ;
• BackupController The object storage service... Will be called , for example ,AWS S3 - Upload backup file . By default ,velero backup create Supports disk snapshots of any persistent volume , You can adjust the snapshot by specifying other flags , function velero backup create --help You can view the available flags , You can also use --snapshot-volumes=false Option to disable snapshots .

About backup storage locations and volume snapshots ,Velero There are two custom resources BackupStorageLocation and VolumeSnapshotLocation, Used for configuration Velero Storage location of the backup and its associated persistent volume snapshots .

• BackupStorageLocation The main back-end storage support is S3 Compatible storage , Store all Velero The prefix in the data store and a set of other provider specific fields . such as ：Minio And Alibaba cloud OSS etc. ;
• VolumeSnapshotLocation（pv data ）, Mainly for PV Take a snapshot , Need plug-ins from cloud providers , Specific fields provided entirely by the provider （ for example AWS Area ,Azure Resource group ,Portworx Snapshot type, etc ） Definition . Take the database and middleware that are most sensitive to data consistency as an example , Open source storage plug-ins Carina Database aware velero Volume snapshot function , It can realize fast backup and recovery of middleware data .

Velero Installation and use

install velero client

$ wget https://mirror.ghproxy.com/https://github.com/vmware-tanzu/velero/releases/download/v1.6.3/velero-v1.6.3-darwin-amd64.tar.gz 
$ tar -zxvf velero-v1.6.3-darwin-amd64.tar.gz && cd velero-v1.6.3-darwin-amd64 
$ mv velero /usr/local/bin && chmod +x /usr/local/bin/velero 
$ velero version

install minio Back end for data backup

Minio install Yaml The documents are as follows ：

apiVersion: v1 
kind: Namespace 
metadata: 
  name: velero 
--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  namespace: velero 
  name: minio 
  labels: 
    component: minio 
spec: 
  strategy: 
    type: Recreate 
  selector: 
    matchLabels: 
      component: minio 
  template: 
    metadata: 
      labels: 
        component: minio 
    spec: 
      volumes: 
      - name: storage 
        emptyDir: {} 
      - name: config 
        emptyDir: {} 
      containers: 
      - name: minio 
        image: minio/minio:latest 
        imagePullPolicy: IfNotPresent 
        args: 
        - server 
        - /storage 
        - --config-dir=/config 
        - --console-address=:9001 
        env: 
        - name: MINIO_ACCESS_KEY 
          value: "minio" 
        - name: MINIO_SECRET_KEY 
          value: "minio123" 
        ports: 
        - containerPort: 9000 
        - containerPort: 9001 
        volumeMounts: 
        - name: storage 
          mountPath: "/storage" 
        - name: config 
          mountPath: "/config" 
--- 
apiVersion: v1 
kind: Service 
metadata: 
  namespace: velero 
  name: minio 
  labels: 
    component: minio 
spec: 
  type: NodePort 
  ports: 
    - name: api 
      port: 9000 
      targetPort: 9000 
    - name: console 
      port: 9001 
      targetPort: 9001 
  selector: 
    component: minio 
--- 
apiVersion: batch/v1 
kind: Job 
metadata: 
  namespace: velero 
  name: minio-setup 
  labels: 
    component: minio 
spec: 
  template: 
    metadata: 
      name: minio-setup 
    spec: 
      restartPolicy: OnFailure 
      volumes: 
      - name: config 
        emptyDir: {} 
      containers: 
      - name: mc 
        image: minio/mc:latest 
        imagePullPolicy: IfNotPresent 
        command: 
        - /bin/sh 
        - -c 
        - "mc --config-dir=/config config host add velero http://minio:9000 minio minio123 && mc --config-dir=/config mb -p velero/velero" 
        volumeMounts: 
        - name: config 
          mountPath: "/config" 

install Mini, And check the resource creation .

$ kubectl apply -f ./00-minio-deployment.yaml 
$ kubectl get pods -n velero 
NAME                     READY   STATUS              RESTARTS   AGE
minio-58dc5cf789-z2777   0/1     ContainerCreating   0          14s
minio-setup-dz4jb        0/1     ContainerCreating   0          6s
$ kubectl get svc  -n velero 
NAME    TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)                         AGE
minio   NodePort   10.96.13.35   <none>        9000:30693/TCP,9001:32351/TCP   17s

After the service has been started , You can log in. minio see velero/velero Of bucket Whether to create successfully .

install velero Server side , Use s3 As the storage

• establish minio voucher

$ cat > credentials-velero <<EOF
[default]
aws_access_key_id = minio
aws_secret_access_key = minio123
EOF
#  install velero
$ cp velero /usr/bin/
#  Enable fast completion 
$ velero completion bash

• Use the official restic Component backup pv

$ velero install    \
   --image velero/velero:v1.6.3  \
  --plugins velero/velero-plugin-for-aws:v1.0.0  \
  --provider aws   \
  --bucket velero   \
  --namespace velero  \
  --secret-file ./credentials-velero   \
  --velero-pod-cpu-request 200m   \
  --velero-pod-mem-request 200Mi   \
  --velero-pod-cpu-limit 1000m  \
  --velero-pod-mem-limit 1000Mi   \
  --use-volume-snapshots=false   \
  --use-restic   \
  --restic-pod-cpu-request 200m   \
  --restic-pod-mem-request 200Mi   \
  --restic-pod-cpu-limit 1000m  \
  --restic-pod-mem-limit 1000Mi  \
  --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000

among , Several important parameters and their descriptions are as follows ：

--provider： The statement uses  Velero  Plug in type .
--plugins： Use  S3 API  Compatible plug-ins  “velero-plugin-for-aws ”.
--bucket： Tencent's cloud  COS  Created bucket name .
--secret-file： visit  COS  Access credentials file for , See created above  “credentials-velero” Voucher file .
--use-restic： Use open source free backup tools  restic  Backup and restore persistent volume data .
--default-volumes-to-restic： Use  restic  To back up everything Pod volume , The premise is that you need to turn on  --use-restic  Parameters .
--backup-location-config： Backup bucket access related configuration .
--region： compatible  S3 API  Of  COS  The bucket area , For example, if the founding area is Guangzhou ,region  Parameter values for “ap-guangzhou”.
--s3ForcePathStyle： Use  S3  File path format .
--s3Url：COS  Compatible  S3 API  Access address 
--use-volume-snapshots=false  To turn off the snapshot backup of storage volume data .

After the installation command is executed , wait for Velero and restic When the workload is ready , Check if the configured storage location is available .

$ velero backup-location get 

apiVersion: velero.io/v1 
kind: BackupStorageLocation 
metadata: 
  name: default 
  namespace: velero 
spec: 
#  Only  aws gcp azure 
  provider: aws 
  objectStorage: 
    bucket: myBucket 
    prefix: backup 
  config: 
    region: us-west-2  
    profile: "default" 
    s3ForcePathStyle: "false" 
    s3Url: http://minio:9000

thus velero It's all deployed .

velero Function is introduced

Create a backup

velero Backup of all objects is supported , Or by type , Namespace and / Or label filtering objects

$ velero create backup $NAME [flags]
$ velero backup create pvc-backup-1  --snapshot-volumes --include-namespaces nginx-example --default-volumes-to-restic --volume-snapshot-locations default

among ：

--include-namespaces: Back up all resources under the namespace , Cluster resources are not included

--include-resources: Type of resource to back up

--include-cluster-resources: Whether to back up cluster resources This option can have three possible values ： true： Include all cluster wide resources ; false： Cluster wide resources are not included ; nil （“ Automatically ” Or not ）

--selector: Select the matching resource backup through the tag

--exclude-namespaces: Resources under this namespace will not be backed up during backup

--exclude-resources: This type of resource will not be backed up during backup

http://--velero.io/exclude-from-backup=true: When the tag selector matches the resource , If the resource has this tag , No backup

meanwhile , You can also use –ordered-resources Parameters , Back up specific kinds of resources in a specific order , You need to specify the resource name and the object name list of the resource , Resource object names are separated by commas , Its name format is “ Namespace / Resource name ”, For cluster wide resources , Just use the resource name . Key value pairs in the map are separated by semicolons , Resource types are plural .

$ velero backup create backupName --include-cluster-resources=true --ordered-resources 'pods=ns1/pod1,ns1/pod2;persistentvolumes=pv4,pv8' --include-namespaces=ns1

$ velero backup create backupName --ordered-resources 'statefulsets=ns1/sts1,ns1/sts0' --include-namespaces=n

Scheduled backup ：

$ velero schedule create <SCHEDULE NAME> --schedule "0 7 * * *"

$ velero create schedule NAME --schedule="@every 6h"

$ velero create schedule NAME --schedule="@every 24h" --include-namespaces web

$ velero create schedule NAME --schedule="@every 168h" --ttl 2160h0m0s

Examples of backup advanced usage

• In a single Velero Create snapshots of more than one persistent volume in the backup

$ velero snapshot-location create ebs-us-east-1 \
    --provider aws \
    --config region=us-east-1

$ velero snapshot-location create portworx-cloud \
    --provider portworx \
    --config type=cloud

$ velero backup create full-cluster-backup \
    --volume-snapshot-locations ebs-us-east-1,portworx-cloud    

• Store backups in different object buckets in different regions

$ velero backup-location create default \
    --provider aws \
    --bucket velero-backups \
    --config region=us-east-1

$ velero backup-location create s3-alt-region \
    --provider aws \
    --bucket velero-backups-alt \
    --config region=us-west-1

$ velero backup create full-cluster-alternate-location-backup \
    --storage-location s3-alt-region

• For storage volumes provided by the public cloud , Store some snapshots locally , Some are stored in the public cloud

$ velero snapshot-location create portworx-local \
    --provider portworx \
    --config type=local

$ velero snapshot-location create portworx-cloud \
    --provider portworx \
    --config type=cloud

$ velero backup create cloud-snapshot-backup \
    --volume-snapshot-locations portworx-cloud    

• Use storage location

$ velero backup-location create default \
    --provider aws \
    --bucket velero-backups \
    --config region=us-west-1

$ velero snapshot-location create ebs-us-west-1 \
    --provider aws \
    --config region=us-west-1

$ velero backup create full-cluster-backup

View backup tasks .

When the backup task status is “Completed” , And the number of errors is 0 , Indicates that the backup task is completed without any errors , You can query through the following command ：

 $ velero backup get 

By temporarily updating the backup storage location to read-only mode , It can prevent the creation or deletion of backup objects in the backup storage location during the restore process .

$ kubectl patch backupstoragelocation default --namespace velero \
    --type merge \
    --patch '{"spec":{"accessMode":"ReadOnly"}}'
    
velero backup-location get
NAME      PROVIDER   BUCKET/PREFIX   PHASE     LAST VALIDATED   ACCESS MODE   DEFAULT
default   aws        velero          Unknown   Unknown          ReadWrite     true

Restore backup data

$ velero restore create --from-backup <backup-name>
$ velero  restore create --from-backup pvc-backup-1 --restore-volumes

View recovery tasks .

$ velero restore get 

When the restore is complete , Don't forget to restore the backup storage location to read-write mode , For the next backup task ：

$ kubectl patch backupstoragelocation default --namespace velero \
   --type merge \
   --patch '{"spec":{"accessMode":"ReadWrite"}}'

Backup hooks Introduce

Velero Support to execute some preset commands in the container before and after the backup task is executed , This method is very effective for data consistency .velero Two methods are supported to specify the hook , One is pod Own annotation declaration , The other is to define Backup During the mission Spec In a statement .

• Pre hooks

pre.hook.backup.velero.io/container: The container that will execute the command , The default is pod The first container in , Optional .

pre.hook.backup.velero.io/command: Commands to execute , If more than one parameter is required , Please specify this command as JSON Array . for example ：["/usr/bin/uname", "-a"]

pre.hook.backup.velero.io/on-error: How to handle if the command returns a non-zero exit code . The default is “Fail”, Valid values are “Fail” and “Continue”, Optional .

pre.hook.backup.velero.io/timeout: Time to wait for the command to execute , If the command exceeds the timeout , It is considered that the hook fails . The default is 30 second , Optional .

• Post hooks

post.hook.backup.velero.io/container: The container that will execute the command , The default is pod The first container in , Optional .

post.hook.backup.velero.io/command: Commands to execute , If more than one parameter is required , Please specify this command as JSON Array . for example ：["/usr/bin/uname", "-a"]

post.hook.backup.velero.io/on-error: How to handle if the command returns a non-zero exit code . The default is “Fail”, Valid values are “Fail” and “Continue”, Optional .

post.hook.backup.velero.io/timeout: Time to wait for the command to execute , If the command exceeds the timeout , It is considered that the hook fails . The default is 30 second , Optional 

Restore hooks Introduce

Velero Support restore hooks, Custom actions that can be performed before or after the restore task . There are two forms of definition ：

• InitContainer Restore Hooks： These will be restored in Pod The application container for will be init The container is added to the restored pod in , To perform any necessary settings .

init.hook.restore.velero.io/container-image: To add init Container image of container 

init.hook.restore.velero.io/container-name: To add init Name of the container 

init.hook.restore.velero.io/command: The task or command to be executed in the initialization container 

Such as before backup , Use the following command to add comments to Pod：

kubectl annotate pod -n <POD_NAMESPACE> <POD_NAME> \
    init.hook.restore.velero.io/container-name=restore-hook \
    init.hook.restore.velero.io/container-image=alpine:latest \
    init.hook.restore.velero.io/command='["/bin/ash", "-c", "date"]'

• Exec Restore Hooks： Can be used in restored Kubernetes pod Execute custom commands or scripts in the container of .

post.hook.restore.velero.io/container:; perform hook The name of the container , The default is the first container , Optional 

post.hook.restore.velero.io/command: Commands to be executed in the container , Required 

post.hook.restore.velero.io/on-error: How to handle execution failure , Valid values are Fail and Continue, The default is Continue, Use Continue Pattern , Only record execution failures ; Use Fail Mode time , Will not be on their own other hook, The restored state will be PartiallyFailed, Optional 

post.hook.restore.velero.io/exec-timeout: How long to wait after starting execution , The default is 30 second , Optional 

post.hook.restore.velero.io/wait-timeout: Time to wait for the container to be ready , This time should be long enough , To enable the container to start , and 

Such as before backup , Use the following command to add comments to Pod

kubectl annotate pod -n <POD_NAMESPACE> <POD_NAME> \
    post.hook.restore.velero.io/container=postgres \
    post.hook.restore.velero.io/command='["/bin/bash", "-c", "psql < /backup/backup.sql"]' \
    post.hook.restore.velero.io/wait-timeout=5m \
    post.hook.restore.velero.io/exec-timeout=45s \
    post.hook.restore.velero.io/on-error=Continue

Velero Analysis of some key problems

Velero Can resources be restored to a namespace different from their backup source ？

Yes , have access to --namespace-mappings Parameter to specify ：

velero restore create RESTORE_NAME \
  --from-backup BACKUP_NAME \
  --namespace-mappings old-ns-1:new-ns-1,old-ns-2:new-ns-2

After the restore operation , Existing NodePort Type of service How to deal with it ？

Velero There is a parameter , Allows the user to decide to keep the original nodePorts.

velero restore create The subcommand has --preserve-nodeports Sign protection services nodePorts. This flag is used to preserve the original... From the backup nodePorts, It can be used as --preserve-nodeports or --preserve-nodeports=true If this flag is given , be Velero In restoring Service Will not delete nodePorts, Instead, try to use what was written during the backup nodePorts.

velero How to implement a consistent backup strategy without affecting the business , And upload the backup data to the object storage ?

If it is based on velero Achieve database consistency , Need to use velero Of hook, The database is backed up before quiesce operation , Backup complete unquiesce. For the backup itself , have access to restic Come on copy data （ But no snapshots ）, Or use snapshots .