Introduction and use of dumping

Dumpling Architecture and features

1. Support multiple data forms ,SQL/CSV
2. Logical derivation
3. Support table filtering , Data row filtering ;
4. Support Amazon S3 Cloud disk
5. in the light of TiDB To optimize ： Support memory limitation , adjustment GC Time , Optimize export performance with hidden columns ;

Dumpling Applicable scenario

1. The amount of exported data is small , A dozen G The best ;
2. Need to export SQL perhaps CSV scene , Data migration can be performed in heterogeneous databases
3. The export efficiency is not high , Because the number is from TiDB Node export , Data reading and conversion are required , Less efficient than physical export ;

Dumpling Not applicable to the scene

1. You need to export directly SST（KEY-VALUE）
2. Incremental backup ： At present, only the full amount is supported , Incremental is not supported ;
3. Scenarios with large amount of data

Deploy

• tiup install dumpling perhaps tidb-toolkit package

jurisdiction

1. SELECT
2. RELOAD
3. LOCK TABLES
4. REPLICATION CLIENT
    

Parameter Introduction

Dumpling export sql
dumpling -u user -P 4000 -h 127.0.0.1 
	--filetype csv  Specify file type ,sql/csv
	--threads 32 	 Concurrency number 
	-o /export/servers/TiDB/backup    Storage directory 
	-r 200000 	 Export time , Maximum lines per file 
	-F 256MiB	 Maximum size of a single file ,KB,MiB,GiB;
	--where "id < 10000"	# All exported tables will be id Screening , Not suitable for the whole library ; 
	--filter "employees.*"	# Export the specified library 
	--filter "*.order"		# Export... Under all libraries order surface ;

Be careful

1. Dummpling The system library will be automatically excluded during full database backup

Consistency assurance

Dumpling adopt --consistency <consistency level> Flag controls export data “ Consistency assurance ” The way

• flush： The backup will be performed before flush table with read lock, The whole cluster can read or write ;FTRL, Warm preparation
• snapshot： The default value is , You can specify the timestamp of the backup , All data to be backed up is the data version at this point in time ;
• lock： Only the backed up tables are locked
• none： Don't lock ; Data changes in real time
• auto： The default method will be selected according to the target database type ,tidb Can choose snapshot,mysql Will use flush

 Dumpling Backup

dumpling]# dumpling -h XX.XX.XX.XX -P4000 -u root --password '' --filetype sql  --threads 2 -o /export/servers/TiDB/backup/dumpling -r 10000 
Release version: v5.0.1
Git commit hash: 4cb115746bb658b6d1a12c0e49932bfd3a08afac
Git branch:      heads/refs/tags/v5.0.1
Build timestamp: 2021-04-23 06:01:59Z
Go version:      go version go1.13 linux/amd64

[2022/02/11 18:08:54.706 +08:00] [INFO] [versions.go:55] ["Welcome to dumpling"] ["Release Version"=v5.0.1] ["Git Commit Hash"=4cb115746bb658b6d1a12c0e49932bfd3a08afac] ["Git Branch"=heads/refs/tags/v5.0.1] ["Build timestamp"="2021-04-23 06:01:59"] ["Go Version"="go version go1.13 linux/amd64"]
[2022/02/11 18:08:54.709 +08:00] [INFO] [config.go:599] ["detect server type"] [type=TiDB]
[2022/02/11 18:08:54.709 +08:00] [INFO] [config.go:618] ["detect server version"] [version=5.3.0]
[2022/02/11 18:08:54.720 +08:00] [INFO] [client.go:193] ["[pd] create pd client with endpoints"] [pd-address="[XX.XX.XX.XX:2379,XX.XX.XX.XX:2379,XX.XX.XX.XX:2379]"]
[2022/02/11 18:08:54.724 +08:00] [INFO] [base_client.go:296] ["[pd] update member urls"] [old-urls="[http://XX.XX.XX.XX:2379,http://XX.XX.XX.XX:2379,http://XX.XX.XX.XX:2379]"] [new-urls="[http://XX.XX.XX.XX:2379,http://XX.XX.XX.XX:2379,http://XX.XX.XX.XX:2379]"]
[2022/02/11 18:08:54.724 +08:00] [INFO] [base_client.go:308] ["[pd] switch leader"] [new-leader=http://XX.XX.XX.XX:2379] [old-leader=]
[2022/02/11 18:08:54.724 +08:00] [INFO] [base_client.go:112] ["[pd] init cluster id"] [cluster-id=7058541070281534107]
[2022/02/11 18:08:54.725 +08:00] [INFO] [dump.go:936] ["generate dumpling gc safePoint id"] [id=dumpling_1644574134725790545]
[2022/02/11 18:08:54.730 +08:00] [INFO] [dump.go:83] ["begin to run Dump"] [conf="{\"s3\":{\"endpoint\":\"\",\"region\":\"\",\"storage-class\":\"\",\"sse\":\"\",\"sse-kms-key-id\":\"\",\"acl\":\"\",\"access-key\":\"\",\"secret-access-key\":\"\",\"provider\":\"\",\"force-path-style\":true,\"use-accelerate-endpoint\":false},\"gcs\":{\"endpoint\":\"\",\"storage-class\":\"\",\"predefined-acl\":\"\",\"credentials-file\":\"\"},\"AllowCleartextPasswords\":false,\"SortByPk\":true,\"NoViews\":true,\"NoHeader\":false,\"NoSchemas\":false,\"NoData\":false,\"CompleteInsert\":false,\"TransactionalConsistency\":true,\"EscapeBackslash\":true,\"DumpEmptyDatabase\":true,\"PosAfterConnect\":false,\"CompressType\":0,\"Host\":\"XX.XX.XX.XX\",\"Port\":4000,\"Threads\":2,\"User\":\"root\",\"Security\":{\"CAPath\":\"\",\"CertPath\":\"\",\"KeyPath\":\"\"},\"LogLevel\":\"info\",\"LogFile\":\"\",\"LogFormat\":\"text\",\"OutputDirPath\":\"/export/servers/TiDB/backup/dumpling\",\"StatusAddr\":\":8281\",\"Snapshot\":\"431115241965748232\",\"Consistency\":\"snapshot\",\"CsvNullValue\":\"\\\\N\",\"SQL\":\"\",\"CsvSeparator\":\",\",\"CsvDelimiter\":\"\\\"\",\"Databases\":[],\"Where\":\"\",\"FileType\":\"sql\",\"ServerInfo\":{\"ServerType\":3,\"ServerVersion\":\"5.3.0\"},\"Rows\":10000,\"ReadTimeout\":900000000000,\"TiDBMemQuotaQuery\":0,\"FileSize\":0,\"StatementSize\":1000000,\"SessionParams\":{\"tidb_snapshot\":\"431115241965748232\"},\"Tables\":null}"]
[2022/02/11 18:08:55.041 +08:00] [WARN] [writer.go:230] ["no data written in table chunk"] [database=tony] [table=hero] [chunkIdx=0]
[2022/02/11 18:08:55.198 +08:00] [WARN] [writer.go:230] ["no data written in table chunk"] [database=tony] [table=hero2] [chunkIdx=0]
[2022/02/11 18:08:55.586 +08:00] [INFO] [collector.go:212] ["backup Success summary: total backup ranges: 7, total success: 7, total failed: 0, total take(backup time): 823.020183ms, total take(real time): 823.083425ms, total size(MB): 5.18, avg speed(MB/s): 6.30, total rows: 262153"]
[2022/02/11 18:08:55.586 +08:00] [INFO] [main.go:81] ["dump data successfully, dumpling will exit now"]

Dumpling Backup file description

.
|-- metadata					// Metadata file , Contains the backup time ,BINLOG POS,GTID etc. 
|-- tony.hero.000000001.sql		// Data files 
|-- tony.hero2.000000001.sql	
|-- tony.hero2-schema.sql		// Create table statement 
|-- tony.hero-schema.sql		
`-- tony-schema-create.sql		// Build library statement 

Other backup methods

Back up the specified library ：

dumpling -h XX.XX.XX.XX -P4000 -u root --password 'xxxxxx' --filetype sql -B tony --threads 2 -o /export/servers/TiDB/backup/dumpling -r 10000 

Back up specific tables ：

dumpling -h XX.XX.XX.XX -P4000 -u root --password 'xxxxxx' --filetype sql -T tony.hero2 --threads 2 -o /export/servers/TiDB/backup/dumpling -r 10000 

Only the backup table name contains hero Of

dumpling -h XX.XX.XX.XX -P4000 -u root --password 'xxxxxx' --filetype sql -f "*.hero*" --threads 2 -o /export/servers/TiDB/backup/dumpling -r 10000 

Filter some data rows

dumpling -h XX.XX.XX.XX -P4000 -u root --password 'xxxxxx' --filetype sql -f "*.hero*" --where "id < 10" --threads 2 -o /export/servers/TiDB/backup/dumpling -r 10000