当前位置:网站首页>Restore backup data on S3 compatible storage with tidb lightning
Restore backup data on S3 compatible storage with tidb lightning
2022-07-06 08:03:00 【Tianxiang shop】
This document describes how to put Kubernetes through TiDB Operator The backup data is restored to TiDB colony .
The recovery method used in this article is based on TiDB Operator v1.1 And above CustomResourceDefinition (CRD) Realization , The bottom layer uses TiDB Lightning TiDB-backend To recover data .
TiDB Lightning It is a high-speed import of full data to TiDB Clustering tools , Available from local disk 、Google Cloud Storage (GCS) or Amazon S3 Cloud disk reads data . at present ,TiDB Lightning Support three kinds of backend :Importer-backend
、Local-backend
、TiDB-backend
. The method introduced in this article uses TiDB-backend
. About the differences and choices of these three backend , see also TiDB Lightning file . If you want to use Importer-backend
perhaps Local-backend
Import data , see also Use TiDB Lightning Import cluster data .
The following examples will be compatible S3 The storage ( Specify the path ) The backup data on is restored to TiDB colony .
Use scenarios
If you need from compatible S3 Store and export backup data to TiDB colony , And have the following requirements for data recovery , You can use the recovery scheme introduced in this article :
- We hope to recover with low resource utilization and low network bandwidth utilization , And it's acceptable 50 GB/ Hours of recovery
- It is required to meet ACID
- During backup TiDB The cluster can still provide services
Preparation before recovery
Before data recovery , You need to prepare to restore the environment , And have the relevant permissions of the database .
Prepare to restore the environment
Download the file backup-rbac.yaml, And execute the following command in
test2
This namespace Create the required for recovery RBAC Related resources :kubectl apply -f backup-rbac.yaml -n test2
Remote storage access authorization .
If from Amazon S3 Recover cluster data , There are three ways to grant permissions , Reference resources AWS Account Authorization Authorized access is compatible S3 Remote storage of ; Use Ceph When testing recovery as back-end storage , It's through AccessKey and SecretKey Mode Authorization , Please refer to adopt AccessKey and SecretKey to grant authorization .
establish
restore-demo2-tidb-secret
secret, The secret Store to access TiDB Clustered root Account and key :kubectl create secret generic restore-demo2-tidb-secret --from-literal=user=root --from-literal=password=${password} --namespace=test2
Get the required database permissions
Use TiDB Lightning take Amazon S3 Restore the backup data on to TiDB Before cluster , Make sure you have the following permissions to back up the database :
jurisdiction | Scope |
---|---|
SELECT | Tables |
INSERT | Tables |
UPDATE | Tables |
DELETE | Tables |
CREATE | Databases, tables |
DROP | Databases, tables |
ALTER | Tables |
Restore the specified backup data to TiDB colony
Be careful
because rclone
There is problem , If you use Amazon S3 Store backup , also Amazon S3 Open the AWS-KMS
encryption , You need yaml Add the following to the file spec.s3.options
Configure to ensure successful backup recovery :
spec: ... s3: ... options: - --ignore-checksum
This section provides a variety of ways to store access . Just use the method that suits your situation .
- adopt AccessKey and SecretKey The way of authorization is determined by Ceph How to recover data
- adopt AccessKey and SecretKey The way of authorization is from Amazon S3 How to recover data
- By binding IAM And Pod The way from Amazon S3 How to recover data
- By binding IAM And ServiceAccount The way from Amazon S3 How to recover data
establish Restore customer resource (CR), Restore the specified backup data to TiDB colony .
Method 1: establish Restore custom resource (CR), adopt AccessKey and SecretKey Authorize the specified backup data by Ceph Restore to TiDB colony .
kubectl apply -f restore.yaml
restore.yaml
The contents of the document are as follows :--- apiVersion: pingcap.com/v1alpha1 kind: Restore metadata: name: demo2-restore namespace: test2 spec: backupType: full to: host: ${tidb_host} port: ${tidb_port} user: ${tidb_user} secretName: restore-demo2-tidb-secret s3: provider: ceph endpoint: ${endpoint} secretName: s3-secret path: s3://${backup_path} # storageClassName: local-storage storageSize: 1Gi
Method 2: establish Restore custom resource (CR), adopt AccessKey and SecretKey Authorize the specified backup data by Amazon S3 Restore to TiDB colony .
kubectl apply -f restore.yaml
restore.yaml
The contents of the document are as follows :--- apiVersion: pingcap.com/v1alpha1 kind: Restore metadata: name: demo2-restore namespace: test2 spec: backupType: full to: host: ${tidb_host} port: ${tidb_port} user: ${tidb_user} secretName: restore-demo2-tidb-secret s3: provider: aws region: ${region} secretName: s3-secret path: s3://${backup_path} # storageClassName: local-storage storageSize: 1Gi
Method 3: establish Restore custom resource (CR), adopt IAM binding Pod Restore the specified backup data to TiDB colony .
kubectl apply -f restore.yaml
restore.yaml
The contents of the document are as follows :--- apiVersion: pingcap.com/v1alpha1 kind: Restore metadata: name: demo2-restore namespace: test2 annotations: iam.amazonaws.com/role: arn:aws:iam::123456789012:role/user spec: backupType: full to: host: ${tidb_host} port: ${tidb_port} user: ${tidb_user} secretName: restore-demo2-tidb-secret s3: provider: aws region: ${region} path: s3://${backup_path} # storageClassName: local-storage storageSize: 1Gi
Method 4: establish Restore custom resource (CR), adopt IAM binding ServiceAccount Restore the specified backup data to TiDB colony .
kubectl apply -f restore.yaml
restore.yaml
The contents of the document are as follows :--- apiVersion: pingcap.com/v1alpha1 kind: Restore metadata: name: demo2-restore namespace: test2 spec: backupType: full serviceAccount: tidb-backup-manager to: host: ${tidb_host} port: ${tidb_port} user: ${tidb_user} secretName: restore-demo2-tidb-secret s3: provider: aws region: ${region} path: s3://${backup_path} # storageClassName: local-storage storageSize: 1Gi
Create good
Restore
CR after , You can view the status of the recovery through the following command :kubectl get rt -n test2 -owide
The above examples will be compatible S3 The storage (spec.s3.path
Under the path ) The backup data in is restored to TiDB colony spec.to.host
. About compatibility S3 Stored configuration items , You can refer to S3 Field is introduced .
more Restore
CR For detailed explanation of fields, please refer to Restore CR Field is introduced .
Be careful
TiDB Operator Will create a PVC, For data recovery , The backup data will be downloaded from the remote storage to PV, And then recover . If you want to delete this after the recovery PVC, You can refer to Delete resources First restore Pod Delete , Then take it. PVC Delete .
边栏推荐
- Nft智能合约发行,盲盒,公开发售技术实战--拼图篇
- Database basic commands
- 1204 character deletion operation (2)
- [redis] Introduction to NoSQL database and redis
- Epoll and IO multiplexing of redis
- P3047 [USACO12FEB]Nearby Cows G(树形dp)
- Notes on software development
- Migrate data from SQL files to tidb
- Introduction to number theory (greatest common divisor, prime sieve, inverse element)
- [Yugong series] February 2022 U3D full stack class 011 unity section 1 mind map
猜你喜欢
IP lab, the first weekly recheck
It's hard to find a job when the industry is in recession
24. Query table data (basic)
"Designer universe" APEC design +: the list of winners of the Paris Design Award in France was recently announced. The winners of "Changsha world center Damei mansion" were awarded by the national eco
[untitled]
[research materials] 2022 enterprise wechat Ecosystem Research Report - Download attached
在 uniapp 中使用阿里图标
[research materials] 2021 China online high growth white paper - Download attached
DataX self check error /datax/plugin/reader/_ drdsreader/plugin. Json] does not exist
Uibehavior, a comprehensive exploration of ugui source code
随机推荐
好用的TCP-UDP_debug工具下载和使用
49. Sound card driven article collection
Secure captcha (unsafe verification code) of DVWA range
PHP - Common magic method (nanny level teaching)
IP lab, the first weekly recheck
[factorial inverse], [linear inverse], [combinatorial counting] Niu Mei's mathematical problems
Convolution, pooling, activation function, initialization, normalization, regularization, learning rate - Summary of deep learning foundation
Golang DNS 随便写写
Document 2 Feb 12 16:54
[research materials] 2022 enterprise wechat Ecosystem Research Report - Download attached
Epoll and IO multiplexing of redis
Risk planning and identification of Oracle project management system
Analysis of Top1 accuracy and top5 accuracy examples
A Closer Look at How Fine-tuning Changes BERT
How to prevent Association in cross-border e-commerce multi account operations?
数据治理:元数据管理篇
"Designer universe" APEC design +: the list of winners of the Paris Design Award in France was recently announced. The winners of "Changsha world center Damei mansion" were awarded by the national eco
MEX有关的学习
ESP系列引脚说明图汇总
Flash return file download