Tidb data migration tool overview

TiDB It provides a wealth of tools related to data migration , For full migration 、 The incremental migration 、 Backup recovery 、 Data synchronization and other scenarios .

This article introduces the scenarios of using these tools 、 Information such as advantages and related limitations . Please choose the right tool according to your needs .

The following figure shows the usage scenarios of each migration tool .

The following table describes the usage scenarios of the migration tool 、 Support upstream and downstream information .

name	Use scenarios	The upstream （ Or input the source file ）	The downstream （ Or output file ）	The main advantage	Usage restriction
TiDB DM	Used to transfer data from to MySQL Protocol compatible databases are migrated to TiDB.	MySQL,MariaDB,Aurora,MySQL	TiDB	Integrated data migration task management tool , Support full migration and incremental synchronization ; Support filtering tables and operations ; Support the consolidation and migration of sub databases and tables .	Recommended for 1TB Stock data migration within .
Dumpling	Used to transfer data from MySQL/TiDB Perform full export .	MySQL,TiDB	SQL,CSV	Support new table-filter, It is more convenient to filter data ; Support export to Amazon S3 Cloud disk	If you plan to export to non TiDB Database recovery , It is recommended to use Dumpling; If it is to another TiDB recovery , It is recommended to use BR.
TiDB Lightning	It is used to import all data into TiDB.	Dumpling Output file ;CSV file ; From local disk or Amazon S3 Cloud disk reads data .	TiDB	Support the rapid import of a large amount of new data , Realize fast initialization TiDB The specified table of the cluster ; Support breakpoint renewal ; Support data filtering .	If you use Local-backend Data import ,TiDB Lightning After operation ,TiDB The cluster will not be able to provide external services normally . If you don't want TiDB The external services of the cluster are affected , You can refer to TiDB Lightning TiDB-backend Data import according to the hardware requirements and deployment methods in .
Backup & Restore (BR)	Through the large amount of data TiDB Cluster for data backup and recovery , Realize data migration .	TiDB	SST;backup.meta file ;backup.lock file	Apply to another TiDB Migrating data . Support data cold backup to external storage , It can be used for disaster recovery .	BR Back to TiCDC / Drainer Upstream cluster , Recovery data cannot be saved by TiCDC / Drainer Sync to downstream .BR Only in new_collations_enabled_on_first_bootstrap Operate between clusters with the same switch value .
TiCDC	Through the pull TiKV The change log implements TiDB Incremental data synchronization tool , It has the ability to restore the data to any upstream location TSO The ability to maintain a consistent state , Support other systems to subscribe to data changes .	TiDB	TiDB,MySQL,Apache Pulsar,Kafka,Confluent	Provide open data protocol (TiCDC Open Protocol).	TiCDC Only tables with at least one valid index can be synchronized . The following scenarios are not supported ： It is not supported to use alone RawKV Of TiKV colony . Temporarily not supported in TiDB Created in SEQUENCE Of DDL Operation and SEQUENCE function .
TiDB Binlog	be used for TiDB Incremental data synchronization between clusters , If one of them TiDB Cluster as another TiDB The slave cluster of the cluster .	TiDB	TiDB,MySQL,Kafka, Incremental backup files	Support real-time backup and recovery . Backup TiDB Cluster data , It can also be used for TiDB Recovery in case of cluster failure .	And parts TiDB Version incompatible , Can't be used together .
sync-diff-inspector	Used to verify MySQL/TiDB Consistency of the two data in .	TiDB,MySQL	TiDB,MySQL	It provides the function of repairing data , It is suitable for repairing a small amount of inconsistent data .	about MySQL and TiDB Data synchronization between does not support online verification . I won't support it JSON、BIT、BINARY、BLOB And so on .

Use TiUP Fast installation

from TiDB 4.0 Start ,TiUP As a package manager , Help you manage easily TiDB Different cluster components in the ecosystem . Now you can only use one TiUP Command line to manage any component .

The first 1 Step ： install TiUP

curl --proto '=https' --tlsv1.2 -sSf https://tiup-mirrors.pingcap.com/install.sh | sh

Redeclare global environment variables

source ~/.bash_profile

The first 2 Step ： Installation of components

You can view all available components through the following command :

tiup list

The following output is for all available components ：

Available components: Name Owner Description ---- ----- ----------- bench pingcap Benchmark database with different workloads br pingcap TiDB/TiKV cluster backup restore tool cdc pingcap CDC is a change data capture tool for TiDB client pingcap Client to connect playground cluster pingcap Deploy a TiDB cluster for production ctl pingcap TiDB controller suite dm pingcap Data Migration Platform manager dmctl pingcap dmctl component of Data Migration Platform errdoc pingcap Document about TiDB errors pd-recover pingcap PD Recover is a disaster recovery tool of PD, used to recover the PD cluster which cannot start or provide services normally playground pingcap Bootstrap a local TiDB cluster for fun tidb pingcap TiDB is an open source distributed HTAP database compatible with the MySQL protocol tidb-lightning pingcap TiDB Lightning is a tool used for fast full import of large amounts of data into a TiDB cluster tiup pingcap TiUP is a command-line component management tool that can help to download and install TiDB platform components to the local system

Select the required components for installation

tiup install dumpling tidb-lightning

Be careful

If you need to install a specific version , have access to  tiup install <component>[:version]  command .

The first 3 Step ： to update TiUP And components ( Optional )

It is recommended to check the update log and compatibility instructions of the new version first

tiup update --self && tiup update dm