Tidb data migration (DM) Introduction

DM It is an integrated migration task management tool , Support from and MySQL Protocol compatible database to TiDB Database migration , Support full load and incremental data transmission , You can filter tables and operations at the same time , You can also merge and migrate databases and tables ;

DM framework
 

dmctl： Used to control DM Command tool of cluster ;

•      User pass dmctl Create 、 to update 、 Delete task ;
•      View the working status of the migration task ;
•      An error occurred while processing the migration task

DM-master Responsible for managing and scheduling data migration tasks ;

•      Save the whole DM Topology information
•      be responsible for DM High availability
•      Monitor the running status of each node
•      Coordinate the entire migration task

DM-worker Responsible for specific data migration tasks ;

•      Responsible for reading mysql The data in it ;
•      Responsible for reading BINLOG, Save to local , Make arrangements ;
•      The arranged BINLOG Send to TiDB Cluster
•      One DM-worker There's only one MySQL Source end
•      Redundant DM-worker Will remain idle ;

Applicable scenario ：

1. From compatible MySQL Database migration synchronization data ;
2. Migration of heterogeneous tables between source and target databases
3. Consolidation and migration of sub database and sub table ;

Deploy

install ： tiup install dm

upgrade ： tiup update --self  && tiup update dm

Generate initialization configuration file ： tiup dm template 

tiup dm deploy ${name} ${version} ./topology.yaml --user root [-p] 

tiup list dm-master     View deployable DM edition
Prompt after successful deployment ： deployed successfully

iup dm deploy dm_default 5.3.0 ./topology.yaml --user root
Starting component `dm`: /root/.tiup/components/dm/v1.7.0/tiup-dm deploy dm_default 5.3.0 ./topology.yaml --user root
...
Cluster `dm_default` deployed successfully, you can start it with command: `tiup dm start dm_default`

View deployed DM: tiup dm list

tiup dm list
Starting component `dm`: /root/.tiup/components/dm/v1.7.0/tiup-dm list
Name        User  Version  Path                                        PrivateKey
----        ----  -------  ----                                        ----------
dm_default  root  v5.3.0   /root/.tiup/storage/dm/clusters/dm_default  /root/.tiup/storage/dm/clusters/dm_default/ssh/id_rsa

View cluster information ：tiup dm display dm_default

Start cluster ：  tiup dm start dm_default

• MySQL The source must be turned on BINLOG
• Table Block& Allow Lists： You can filter the library table ;
• BINLOG EVENT FILTER:  Filter specific operations , Such as  INSERT/DELETE/UPDATE/DROP/TRUNCATE etc. ;
• Table Routing： Table routing , Different table names of source and target can be synchronized ; You can synchronize databases and tables to a single table ;
• TiDB： take BINLOG Turn into TiDB Executable statements , Parallel execution ;

DM The configuration file

# The topology template is used deploy a minimal DM cluster, which suitable
# for scenarios with only three machinescontains. The minimal cluster contains
# - 3 master nodes
# - 3 worker nodes
# You can change the hosts according your environment
---
global:
  user: "root"
  ssh_port: 22
  deploy_dir: "/export/servers/tidb/dm/deploy"
  data_dir: "/export/servers/tidb/dm/data"
  # arch: "amd64"

master_servers:
  - host: XX.XX.XX.XX

worker_servers:
  - host: XX.XX.XX.XX

monitoring_servers:
  - host: XX.XX.XX.XX


grafana_servers:
  - host: XX.XX.XX.XX

alertmanager_servers:
  - host: XX.XX.XX.XX

Data source configuration

source-id: "mysql-replica-01"
# Open or not GTID
enable-gtid: false

#  Open or not  relay log
enable-relay: false
relay-binlog-name: '' # Pull the upstream starting file name 
relay-binlog-gtid: '' # Pull upstream BINLOG The beginning of GTID

from:
        host: "x.x.x.x"
        port: port
        user: "username"
        password: "password"

Target library configuration

#  Target library configuration 
#  Task name ,  Multiple tasks running at the same time cannot have the same name 
name: "test"
#  Total quantity + The incremental : all  Migration patterns 
task-mode: "all"
#  The downstream TiDB Configuration information 
target-database: 
	host: "X.X.X.X"
	port: port
	user: "user"
	password: "password"

Filter rule configuration

mysql-instance:
	- source-id: "mysql-replica-01"		#  from source-id = mysql-replica-01  Data source migration data 
	  block-allow-list: "bw-rule-1"		#  Black and white list configuration name , Filter the specified library table 
	  filter-rules: ["filter-rule-1"]	#  Rules for filtering data source specific actions ,  Multiple filtering rules can be configured 
	  route-rules: ["route-rule-1", "route-rule-2"]	# The data source table is migrated to the target TiDB Table routing rules ,  Multiple rules can be defined 
	- source-id: "mysql-replica-02"		#  from source-id = my-replica-02  Data source migration data 
	  block-allow-list: "bw-rule-2"		#  Black and white list configuration name 
	  filter-rules: ["filter-rule-2"]	#  Rules for filtering data source specific actions ,  Multiple filtering rules can be configured 	  
	  
#  Black and white list configuration ：
block-allow-list:				# Define filtering rules for data source migration tables ,  Multiple rules can be defined .  If DM The version is earlier than v2.0.0-beta.2  Then use 
block-white-list
	bw-rule-1:					#  Rule name 
		do-dbs: ["user"]		#  Which libraries to migrate 
		ignore-dbs: ["test"]	#  Ignore which libraries 
		do-tables:				#  Which tables to migrate 
		- db-name: "user"		#  transfer user In the library t1 surface 
		  tbl-name:  "t1"

# Binlog event filter, Operational filtering 
filters:						#  Table matching the upstream database instance binlog event filter Rule set 
	filter-rule-1:				#  Configuration name 
		schema-pattern: "test"		#  Database name matching rules ,  Support for wildcards  "*"  and  "?"
		table-pattern: "sbtest"		#  Table name matching rules ,  Support for wildcards  "*"  and  "?"
		events: ["insert"]			#  What to match event type 
		action: Ignore				#  For those that meet the matching rules  binlog  transfer (Do) Or ignore (Ignore)
	filter-rule-2:
		schema-pattern: "test"
		events: ["truncate table"]
		action: Ignore
		
#  Routing rules 
routes:								#  The route between upstream and downstream  table routing  Rule set 
	route-rule-1:					#  Configuration name 
		schema-pattern: "test"		#  Database name matching rules , Support for wildcards 
		table-pattern: 	"sbtest1"	#  Table name matching rules , Support for wildcards 
		target-schema: "test"		#  Target library name 
		target-table: "sbtest2"		#  Target table name 
		
#  Merge and migrate databases and tables 
routes:
	sale-route-rule:
		schema-pattern: "sales"
		table-pattern: "order*"
		target-schema: "sales"
		target-table: "order"

Common commands

#  Inspection task 
tiup dmctl --master-addr 172.18.136.172:8261  check-task ./task.yaml

#  Start task 
tiup dmctl --master-addr 172.18.136.172:8261  start-task ./task.yaml

#  Suspend task 
tiup dmctl --master-addr 172.18.136.172:8261  pause-task ./task.yaml

#  Recovery task 
tiup dmctl --master-addr 172.18.136.172:8261  resume-task ./task.yaml

#  View tasks 
tiup dmctl --master-addr 172.18.136.172:8261  query-task ./task.yaml

#  Stop task 
tiup dmctl --master-addr 172.18.136.172:8261  stop-task ./task.yaml

performance optimization

• Full export
  •     rows                # Every time you export chunk Row number ;
  •     chunk-filesize        #   Default 64M
  •     threads             # Number of concurrent threads    
• Full import
  •     pool-size            # TiDB Number of threads imported concurrently , Default 16;
• Incremental replication
  •     worker-count        # Default 16, DM worker Number of threads
  •     batch                # Downstream database , The number of rows each thing contains ; Default 100

common problem

1, Handle incompatible DDL
tiup dmctl --master-addr ip:8261 handle-error test skip

2, Self increasing primary key conflicts ; DM Primary key conflicts may occur in the scenario of merging and migrating databases and tables ;
    a, Remove the primary key attribute of the auto increment primary key
    b,   Use federated primary keys
 

Usage restriction

• Data source version
  •  MySQL Version  > 5.5
  •  MariaDB Version >= 10.1.2
• DDL Pre launch compatibility
  •   at present TiDB Partial compatibility MySQL DDL;
• Sub database and sub table
  • Primary key conflict ; 
• Sync MySQL Instance change of ;
  • Master slave switch occurs , Lead to BINLOG atypism ; You need to manually switch ;