Migrate data from SQL files to tidb

This article describes how to use TiDB Lightning from MySQL SQL File migration data to TiDB. About how to generate MySQL SQL file , Please refer to Dumpling In document Export to SQL file .

Prerequisite

• Use TiUP install TiDB Lightning
• Lightning Required downstream database permissions

The first 1 Step ： Get ready SQL file

Will all SQL Put the files in the unified directory , for example  /data/my_datasource/  or  s3://my-bucket/sql-backup?region=us-west-2.Lightning It will recursively find all... Under this directory and its subdirectories  .sql  file .

The first 2 Step ： Define the target table structure

To import TiDB, It must be for SQL The document provides a table structure .

If you use Dumpling Tools export data , The table structure file will be automatically exported . Besides , Data exported in other ways can be created into a table structure by any of the following methods ：

• Method 1 ： Use TiDB Lightning Create a table structure .

  Write include DDL Of the statement SQL file ：

  • The file name format is  ${db_name}-schema-create.sql, The content should include  CREATE DATABASE  sentence .
  • The file name format is  ${db_name}.${table_name}-schema.sql, The content should include  CREATE TABLE  sentence .

• Method 2 ： Manually downstream TiDB Database and table building .

The first 3 Step ： Writing configuration files

New file  tidb-lightning.toml, Contains the following ：

[lightning] # journal level = "info" file = "tidb-lightning.log" [tikv-importer] # "local"： This mode is used by default , Apply to TiB Large amount of data above level , But downstream during import TiDB Unable to provide external services . backend = "local" # # "tidb"：TiB Data volume below level can also be used `tidb` Back end mode , The downstream TiDB Can normally provide services . For more information about the backend mode, please refer to https://docs.pingcap.com/zh/tidb/stable/tidb-lightning-backends . # Set the temporary storage address of sorted key value pairs , The destination path must be an empty directory , The directory space must be larger than the size of the dataset to be imported . It is recommended to set it to and `data-source-dir` Different disk directories and flash media , Monopoly I/O You will get better import performance . sorted-kv-dir = "${sorted-kv-dir}" [mydumper] # Source data directory data-source-dir = "${data-path}" # Local or S3 route , for example ：'s3://my-bucket/sql-backup?region=us-west-2' [tidb] # Information about the target cluster host = ${host} # for example ：172.16.32.1 port = ${port} # for example ：4000 user = "${user_name}" # for example ："root" password = "${password}" # for example ："rootroot" status-port = ${status-port} # Import process Lightning You need to start from TiDB Of “ Status port ” Get table structure information , for example ：10080 pd-addr = "${ip}:${port}" # colony PD The address of ,Lightning adopt PD Get some information , for example 172.16.31.3:2379. When backend = "local" when status-port and pd-addr Must be filled in correctly , Otherwise, an exception will appear in the import .

More information about configuration files , Please see the  TiDB Lightning Configuration.

The first 4 Step ： Perform import

function  tidb-lightning. If you start the program directly from the command line , Maybe because  SIGHUP  Signal and exit , Suggest cooperation  nohup  or  screen  Tools such as .

If from S3 Import , You need to have access to the Amazon S3 The account number stored in the backend SecretKey and AccessKey Pass in as an environment variable Lightning node .

export AWS_ACCESS_KEY_ID=${access_key} export AWS_SECRET_ACCESS_KEY=${secret_key} nohup tiup tidb-lightning -config tidb-lightning.toml > nohup.out 2>&1 &

meanwhile ,TiDB Lightning Also support from  ~/.aws/credentials  Read the credential file .

After the import starts , You can view the progress in any of the following ways ：

• adopt  grep  Log keywords  progress  View progress , Default 5 Minute update .
• Check the progress through the monitoring panel , Please refer to  TiDB Lightning monitor .
• adopt Web Page view progress , Please refer to  Web Interface .

After import ,TiDB Lightning Will automatically exit . Check the last of the log 5 There will be  the whole procedure completed, It indicates that the import was successful .

Be careful

Whether the import is successful or not , The last line will show  tidb lightning exit. It just means TiDB Lightning The normal exit , Does not mean that the task is completed .