当前位置:网站首页>Migrate data from SQL files to tidb
Migrate data from SQL files to tidb
2022-07-06 08:01:00 【Tianxiang shop】
This article describes how to use TiDB Lightning from MySQL SQL File migration data to TiDB. About how to generate MySQL SQL file , Please refer to Dumpling In document Export to SQL file .
Prerequisite
The first 1 Step : Get ready SQL file
Will all SQL Put the files in the unified directory , for example /data/my_datasource/
or s3://my-bucket/sql-backup?region=us-west-2
.Lightning It will recursively find all... Under this directory and its subdirectories .sql
file .
The first 2 Step : Define the target table structure
To import TiDB, It must be for SQL The document provides a table structure .
If you use Dumpling Tools export data , The table structure file will be automatically exported . Besides , Data exported in other ways can be created into a table structure by any of the following methods :
Method 1 : Use TiDB Lightning Create a table structure .
Write include DDL Of the statement SQL file :
- The file name format is
${db_name}-schema-create.sql
, The content should includeCREATE DATABASE
sentence . - The file name format is
${db_name}.${table_name}-schema.sql
, The content should includeCREATE TABLE
sentence .
- The file name format is
Method 2 : Manually downstream TiDB Database and table building .
The first 3 Step : Writing configuration files
New file tidb-lightning.toml
, Contains the following :
[lightning] # journal level = "info" file = "tidb-lightning.log" [tikv-importer] # "local": This mode is used by default , Apply to TiB Large amount of data above level , But downstream during import TiDB Unable to provide external services . backend = "local" # # "tidb":TiB Data volume below level can also be used `tidb` Back end mode , The downstream TiDB Can normally provide services . For more information about the backend mode, please refer to https://docs.pingcap.com/zh/tidb/stable/tidb-lightning-backends . # Set the temporary storage address of sorted key value pairs , The destination path must be an empty directory , The directory space must be larger than the size of the dataset to be imported . It is recommended to set it to and `data-source-dir` Different disk directories and flash media , Monopoly I/O You will get better import performance . sorted-kv-dir = "${sorted-kv-dir}" [mydumper] # Source data directory data-source-dir = "${data-path}" # Local or S3 route , for example :'s3://my-bucket/sql-backup?region=us-west-2' [tidb] # Information about the target cluster host = ${host} # for example :172.16.32.1 port = ${port} # for example :4000 user = "${user_name}" # for example :"root" password = "${password}" # for example :"rootroot" status-port = ${status-port} # Import process Lightning You need to start from TiDB Of “ Status port ” Get table structure information , for example :10080 pd-addr = "${ip}:${port}" # colony PD The address of ,Lightning adopt PD Get some information , for example 172.16.31.3:2379. When backend = "local" when status-port and pd-addr Must be filled in correctly , Otherwise, an exception will appear in the import .
More information about configuration files , Please see the TiDB Lightning Configuration.
The first 4 Step : Perform import
function tidb-lightning
. If you start the program directly from the command line , Maybe because SIGHUP
Signal and exit , Suggest cooperation nohup
or screen
Tools such as .
If from S3 Import , You need to have access to the Amazon S3 The account number stored in the backend SecretKey and AccessKey Pass in as an environment variable Lightning node .
export AWS_ACCESS_KEY_ID=${access_key} export AWS_SECRET_ACCESS_KEY=${secret_key} nohup tiup tidb-lightning -config tidb-lightning.toml > nohup.out 2>&1 &
meanwhile ,TiDB Lightning Also support from ~/.aws/credentials
Read the credential file .
After the import starts , You can view the progress in any of the following ways :
- adopt
grep
Log keywordsprogress
View progress , Default 5 Minute update . - Check the progress through the monitoring panel , Please refer to TiDB Lightning monitor .
- adopt Web Page view progress , Please refer to Web Interface .
After import ,TiDB Lightning Will automatically exit . Check the last of the log 5 There will be the whole procedure completed
, It indicates that the import was successful .
Be careful
Whether the import is successful or not , The last line will show tidb lightning exit
. It just means TiDB Lightning The normal exit , Does not mean that the task is completed .
边栏推荐
- [research materials] 2021 live broadcast annual data report of e-commerce - Download attached
- Hcip day 16
- Chinese Remainder Theorem (Sun Tzu theorem) principle and template code
- 华为云OBS文件上传下载工具类
- onie支持pice硬盘
- Pangolin Library: control panel, control components, shortcut key settings
- 图像融合--挑战、机遇与对策
- NFT smart contract release, blind box, public offering technology practice -- contract
- [redis] Introduction to NoSQL database and redis
- 灰度升级 TiDB Operator
猜你喜欢
Inspiration from the recruitment of bioinformatics analysts in the Department of laboratory medicine, Zhujiang Hospital, Southern Medical University
[nonlinear control theory]9_ A series of lectures on nonlinear control theory
[untitled]
Nft智能合约发行,盲盒,公开发售技术实战--合约篇
[research materials] 2022 China yuancosmos white paper - Download attached
Document 2 Feb 12 16:54
将 NFT 设置为 ENS 个人资料头像的分步指南
Pangolin Library: control panel, control components, shortcut key settings
Data governance: 3 characteristics, 4 transcendence and 3 28 principles of master data
Secure captcha (unsafe verification code) of DVWA range
随机推荐
07- [istio] istio destinationrule (purpose rule)
PHP Coding Standard
Binary tree creation & traversal
matplotlib. Widgets are easy to use
Description of octomap averagenodecolor function
使用 Dumpling 备份 TiDB 集群数据到兼容 S3 的存储
Learn Arduino with examples
Position() function in XPath uses
Linked list interview questions (Graphic explanation)
Make learning pointer easier (3)
The ECU of 21 Audi q5l 45tfsi brushes is upgraded to master special adjustment, and the horsepower is safely and stably increased to 305 horsepower
链表面试题(图文详解)
Step by step guide to setting NFT as an ens profile Avatar
PHP - Common magic method (nanny level teaching)
Solution: système de surveillance vidéo intelligent de patrouille sur le chantier
Data governance: Data Governance under microservice architecture
A Closer Look at How Fine-tuning Changes BERT
好用的TCP-UDP_debug工具下载和使用
Onie supports pice hard disk
你想知道的ArrayList知识都在这