当前位置:网站首页>Migrate data from SQL files to tidb
Migrate data from SQL files to tidb
2022-07-06 08:01:00 【Tianxiang shop】
This article describes how to use TiDB Lightning from MySQL SQL File migration data to TiDB. About how to generate MySQL SQL file , Please refer to Dumpling In document Export to SQL file .
Prerequisite
The first 1 Step : Get ready SQL file
Will all SQL Put the files in the unified directory , for example /data/my_datasource/
or s3://my-bucket/sql-backup?region=us-west-2
.Lightning It will recursively find all... Under this directory and its subdirectories .sql
file .
The first 2 Step : Define the target table structure
To import TiDB, It must be for SQL The document provides a table structure .
If you use Dumpling Tools export data , The table structure file will be automatically exported . Besides , Data exported in other ways can be created into a table structure by any of the following methods :
Method 1 : Use TiDB Lightning Create a table structure .
Write include DDL Of the statement SQL file :
- The file name format is
${db_name}-schema-create.sql
, The content should includeCREATE DATABASE
sentence . - The file name format is
${db_name}.${table_name}-schema.sql
, The content should includeCREATE TABLE
sentence .
- The file name format is
Method 2 : Manually downstream TiDB Database and table building .
The first 3 Step : Writing configuration files
New file tidb-lightning.toml
, Contains the following :
[lightning] # journal level = "info" file = "tidb-lightning.log" [tikv-importer] # "local": This mode is used by default , Apply to TiB Large amount of data above level , But downstream during import TiDB Unable to provide external services . backend = "local" # # "tidb":TiB Data volume below level can also be used `tidb` Back end mode , The downstream TiDB Can normally provide services . For more information about the backend mode, please refer to https://docs.pingcap.com/zh/tidb/stable/tidb-lightning-backends . # Set the temporary storage address of sorted key value pairs , The destination path must be an empty directory , The directory space must be larger than the size of the dataset to be imported . It is recommended to set it to and `data-source-dir` Different disk directories and flash media , Monopoly I/O You will get better import performance . sorted-kv-dir = "${sorted-kv-dir}" [mydumper] # Source data directory data-source-dir = "${data-path}" # Local or S3 route , for example :'s3://my-bucket/sql-backup?region=us-west-2' [tidb] # Information about the target cluster host = ${host} # for example :172.16.32.1 port = ${port} # for example :4000 user = "${user_name}" # for example :"root" password = "${password}" # for example :"rootroot" status-port = ${status-port} # Import process Lightning You need to start from TiDB Of “ Status port ” Get table structure information , for example :10080 pd-addr = "${ip}:${port}" # colony PD The address of ,Lightning adopt PD Get some information , for example 172.16.31.3:2379. When backend = "local" when status-port and pd-addr Must be filled in correctly , Otherwise, an exception will appear in the import .
More information about configuration files , Please see the TiDB Lightning Configuration.
The first 4 Step : Perform import
function tidb-lightning
. If you start the program directly from the command line , Maybe because SIGHUP
Signal and exit , Suggest cooperation nohup
or screen
Tools such as .
If from S3 Import , You need to have access to the Amazon S3 The account number stored in the backend SecretKey and AccessKey Pass in as an environment variable Lightning node .
export AWS_ACCESS_KEY_ID=${access_key} export AWS_SECRET_ACCESS_KEY=${secret_key} nohup tiup tidb-lightning -config tidb-lightning.toml > nohup.out 2>&1 &
meanwhile ,TiDB Lightning Also support from ~/.aws/credentials
Read the credential file .
After the import starts , You can view the progress in any of the following ways :
- adopt
grep
Log keywordsprogress
View progress , Default 5 Minute update . - Check the progress through the monitoring panel , Please refer to TiDB Lightning monitor .
- adopt Web Page view progress , Please refer to Web Interface .
After import ,TiDB Lightning Will automatically exit . Check the last of the log 5 There will be the whole procedure completed
, It indicates that the import was successful .
Be careful
Whether the import is successful or not , The last line will show tidb lightning exit
. It just means TiDB Lightning The normal exit , Does not mean that the task is completed .
边栏推荐
- [count] [combined number] value series
- Common functions for PHP to process strings
- Nc204382 medium sequence
- Type of data in energy dashboard
- 在 uniapp 中使用阿里图标
- The ECU of 21 Audi q5l 45tfsi brushes is upgraded to master special adjustment, and the horsepower is safely and stably increased to 305 horsepower
- Database basic commands
- 使用 Dumpling 备份 TiDB 集群数据到兼容 S3 的存储
- 2.10transfrom attribute
- 软件开发的一点随记
猜你喜欢
From monomer structure to microservice architecture, introduction to microservices
你想知道的ArrayList知识都在这
在 uniapp 中使用阿里图标
Golang DNS write casually
The ECU of 21 Audi q5l 45tfsi brushes is upgraded to master special adjustment, and the horsepower is safely and stably increased to 305 horsepower
Uibehavior, a comprehensive exploration of ugui source code
The Vice Minister of the Ministry of industry and information technology of "APEC industry +" of the national economic and information technology center led a team to Sichuan to investigate the operat
[research materials] 2021 China online high growth white paper - Download attached
[t31zl intelligent video application processor data]
IP lab, the first weekly recheck
随机推荐
"Designer universe" APEC design +: the list of winners of the Paris Design Award in France was recently announced. The winners of "Changsha world center Damei mansion" were awarded by the national eco
The ECU of 21 Audi q5l 45tfsi brushes is upgraded to master special adjustment, and the horsepower is safely and stably increased to 305 horsepower
TiDB备份与恢复简介
Machine learning - decision tree
Onie supports pice hard disk
链表面试题(图文详解)
MFC sends left click, double click, and right click messages to list controls
Binary tree creation & traversal
The State Economic Information Center "APEC industry +" Western Silicon Valley will invest 2trillion yuan in Chengdu Chongqing economic circle, which will surpass the observation of Shanghai | stable
Yu Xia looks at win system kernel -- message mechanism
[research materials] 2022 China yuancosmos white paper - Download attached
Codeforces Global Round 19(A~D)
matplotlib. Widgets are easy to use
Flash return file download
继电反馈PID控制器参数自整定
Learn Arduino with examples
二叉树创建 & 遍历
使用 Dumpling 备份 TiDB 集群数据到兼容 S3 的存储
CAD ARX gets the current viewport settings
Generator Foundation