Tidb data migration scenario overview

The general introduction of this document can be used for TiDB Data migration solutions for . The data migration scheme is as follows ：

• Full data migration .

  • Data import ： Use TiDB Lightning take Aurora Snapshot,CSV File or Mydumper SQL Import all the data of the file into TiDB colony .
  • Export data ： Use Dumpling take TiDB The total data of the cluster is exported as CSV File or Mydumper SQL file , So as to better cooperate from MySQL A database or MariaDB Database for data migration .
  • TiDB DM (Data migration) It also provides a database suitable for small-scale data volume （ For example, is less than 1 TiB） Full data migration function .

• Quick initialization TiDB colony ：TiDB Lightning The fast import function provided can realize fast initialization TiDB The effect of the specified table of the cluster . Please note that , Use quick initialization TiDB Cluster function pair TiDB The impact of clusters is great , During initialization ,TiDB The cluster does not support external access .

• Incremental data migration ： Use TiDB DM from MySQL,MariaDB or Aurora Sync Binlog To TiDB, This function can greatly reduce the downtime window time in the process of business migration .

• TiDB Cluster replication ：TiDB Support backup and recovery , This function can realize TiDB Initialize a snapshot of to another brand new TiDB colony .

According to the type of database where the migrated data is located 、 Deployment location 、 Business data size 、 Business needs and other factors , There will be different data migration options . Here are some common data migration scenarios , It is convenient for users to choose the most suitable data migration scheme according to these clues .

transfer Aurora MySQL To TiDB

from Aurora Migrate data to deploy in AWS Of TiDB colony , Data migration can be divided into two steps: full migration and incremental migration . Please choose the corresponding steps according to your business needs .

• from Aurora Migrate data to TiDB

transfer MySQL To TiDB

If you don't use it Cloud storage (S3) service , And the network connection and delay are good , So from MySQL Migrate data to TiDB You can refer to the following scheme .

• Small amount of data MySQL Migrate data to TiDB

If you have requirements for data migration speed , Or the data scale is particularly large （ For example, greater than 1 TiB）, And forbid TiDB There are other businesses written to the cluster during the migration , Then you can use Lightning Perform a quick import , Then choose whether to use... According to business needs DM Perform incremental data (Binlog) Sync .

• From the large amount of data MySQL Migrate data to TiDB

Sub database and sub table MySQL Merge and migrate to TiDB

If your business is based on MySQL Sub database scheme to store data , Business data from MySQL Migrate to TiDB after , Merge these sub table data into a consolidated , Then you can use DM Perform split table consolidation and migration .

• Small data volume, sub database and sub table MySQL Merge and migrate data to TiDB

If the total scale of sub table data is particularly large （ For example, greater than 1 TiB）, And forbid TiDB There are other businesses written to the cluster during the migration , Then you can use Lightning Quickly merge and import split table data , Then choose whether to use... According to business needs DM Perform incremental data (Binlog) Table synchronization .

• From the big data volume sub database sub table MySQL Merge and migrate data to TiDB

Migrate data from files to TiDB

• from CSV File migration data to TiDB
• from SQL File migration data to TiDB

Complex migration scenarios

DM During real-time synchronization , Several existing features can make the synchronization process more flexible , Adapt to various business needs ：

• Upstream use pt/gh-ost Continuous synchronization scenario of the tool
• There are more migration scenarios downstream
• How to filter binlog event
• How to use SQL Expression filtering binlog