Restore backup data on S3 compatible storage with tidb lightning

This document describes how to put Kubernetes through TiDB Operator The backup data is restored to TiDB colony .

The recovery method used in this article is based on TiDB Operator v1.1 And above CustomResourceDefinition (CRD) Realization , The bottom layer uses  TiDB Lightning TiDB-backend  To recover data .

TiDB Lightning It is a high-speed import of full data to TiDB Clustering tools , Available from local disk 、Google Cloud Storage (GCS) or Amazon S3 Cloud disk reads data . at present ,TiDB Lightning Support three kinds of backend ：Importer-backend、Local-backend、TiDB-backend. The method introduced in this article uses  TiDB-backend. About the differences and choices of these three backend , see also  TiDB Lightning file . If you want to use  Importer-backend  perhaps  Local-backend  Import data , see also Use TiDB Lightning Import cluster data .

The following examples will be compatible S3 The storage （ Specify the path ） The backup data on is restored to TiDB colony .

Use scenarios

If you need from compatible S3 Store and export backup data to TiDB colony , And have the following requirements for data recovery , You can use the recovery scheme introduced in this article ：

• We hope to recover with low resource utilization and low network bandwidth utilization , And it's acceptable 50 GB/ Hours of recovery
• It is required to meet ACID
• During backup TiDB The cluster can still provide services

Preparation before recovery

Before data recovery , You need to prepare to restore the environment , And have the relevant permissions of the database .

Prepare to restore the environment

1. Download the file  backup-rbac.yaml, And execute the following command in  test2  This namespace Create the required for recovery RBAC Related resources ：

   kubectl apply -f backup-rbac.yaml -n test2

2. Remote storage access authorization .

   If from Amazon S3 Recover cluster data , There are three ways to grant permissions , Reference resources  AWS Account Authorization Authorized access is compatible S3 Remote storage of ; Use Ceph When testing recovery as back-end storage , It's through AccessKey and SecretKey Mode Authorization , Please refer to adopt AccessKey and SecretKey to grant authorization .

3. establish  restore-demo2-tidb-secret secret, The secret Store to access TiDB Clustered root Account and key ：

   kubectl create secret generic restore-demo2-tidb-secret --from-literal=user=root --from-literal=password=${password} --namespace=test2

Get the required database permissions

Use TiDB Lightning take Amazon S3 Restore the backup data on to TiDB Before cluster , Make sure you have the following permissions to back up the database ：

jurisdiction	Scope
SELECT	Tables
INSERT	Tables
UPDATE	Tables
DELETE	Tables
CREATE	Databases, tables
DROP	Databases, tables
ALTER	Tables

Restore the specified backup data to TiDB colony

Be careful

because  rclone  There is problem , If you use Amazon S3 Store backup , also Amazon S3 Open the  AWS-KMS  encryption , You need yaml Add the following to the file  spec.s3.options  Configure to ensure successful backup recovery ：

spec: ... s3: ... options: - --ignore-checksum

This section provides a variety of ways to store access . Just use the method that suits your situation .

• adopt AccessKey and SecretKey The way of authorization is determined by Ceph How to recover data
• adopt AccessKey and SecretKey The way of authorization is from Amazon S3 How to recover data
• By binding IAM And Pod The way from Amazon S3 How to recover data
• By binding IAM And ServiceAccount The way from Amazon S3 How to recover data

1. establish Restore customer resource (CR), Restore the specified backup data to TiDB colony .

   • Method 1： establish Restore custom resource (CR), adopt AccessKey and SecretKey Authorize the specified backup data by Ceph Restore to TiDB colony .

     kubectl apply -f restore.yaml

     restore.yaml  The contents of the document are as follows ：

     --- apiVersion: pingcap.com/v1alpha1 kind: Restore metadata: name: demo2-restore namespace: test2 spec: backupType: full to: host: ${tidb_host} port: ${tidb_port} user: ${tidb_user} secretName: restore-demo2-tidb-secret s3: provider: ceph endpoint: ${endpoint} secretName: s3-secret path: s3://${backup_path} # storageClassName: local-storage storageSize: 1Gi

   • Method 2： establish Restore custom resource (CR), adopt AccessKey and SecretKey Authorize the specified backup data by Amazon S3 Restore to TiDB colony .

     kubectl apply -f restore.yaml

     restore.yaml  The contents of the document are as follows ：

     --- apiVersion: pingcap.com/v1alpha1 kind: Restore metadata: name: demo2-restore namespace: test2 spec: backupType: full to: host: ${tidb_host} port: ${tidb_port} user: ${tidb_user} secretName: restore-demo2-tidb-secret s3: provider: aws region: ${region} secretName: s3-secret path: s3://${backup_path} # storageClassName: local-storage storageSize: 1Gi

   • Method 3： establish Restore custom resource (CR), adopt IAM binding Pod Restore the specified backup data to TiDB colony .

     kubectl apply -f restore.yaml

     restore.yaml  The contents of the document are as follows ：

     --- apiVersion: pingcap.com/v1alpha1 kind: Restore metadata: name: demo2-restore namespace: test2 annotations: iam.amazonaws.com/role: arn:aws:iam::123456789012:role/user spec: backupType: full to: host: ${tidb_host} port: ${tidb_port} user: ${tidb_user} secretName: restore-demo2-tidb-secret s3: provider: aws region: ${region} path: s3://${backup_path} # storageClassName: local-storage storageSize: 1Gi

   • Method 4： establish Restore custom resource (CR), adopt IAM binding ServiceAccount Restore the specified backup data to TiDB colony .

     kubectl apply -f restore.yaml

     restore.yaml  The contents of the document are as follows ：

     --- apiVersion: pingcap.com/v1alpha1 kind: Restore metadata: name: demo2-restore namespace: test2 spec: backupType: full serviceAccount: tidb-backup-manager to: host: ${tidb_host} port: ${tidb_port} user: ${tidb_user} secretName: restore-demo2-tidb-secret s3: provider: aws region: ${region} path: s3://${backup_path} # storageClassName: local-storage storageSize: 1Gi

2. Create good  Restore CR after , You can view the status of the recovery through the following command ：

   kubectl get rt -n test2 -owide

The above examples will be compatible S3 The storage （spec.s3.path  Under the path ） The backup data in is restored to TiDB colony  spec.to.host. About compatibility S3 Stored configuration items , You can refer to  S3 Field is introduced .

more  Restore CR For detailed explanation of fields, please refer to Restore CR Field is introduced .

Be careful

TiDB Operator Will create a PVC, For data recovery , The backup data will be downloaded from the remote storage to PV, And then recover . If you want to delete this after the recovery PVC, You can refer to Delete resources First restore Pod Delete , Then take it. PVC Delete .