TiDB Lightning

TiDB Lightning principle

TiDB Lightning yes TiDB One of the ecological tools , Can be Dumpling Derived sql perhaps csv High speed import of full file data to TiDB In the cluster

1. Import mode ： Will automatically TiKV The cluster switches to import mode , The import mode optimizes writes , And stop automatic data compression , So as to improve the efficiency of import ; This will affect the read / write performance of online services ;
2. establish schema And table
3. Split table
4. Read SQL dump
5. Write local temporary storage file ： Convert the data source to TiKV Key value pairs with the same encoding method , Sort and store in TiKV In the local storage of ;
6. Import to TiKV colony ： Load into TiKV Of RocksDB in
7. Verification and analysis
8. Switch to normal mode ： If the import process is interrupted abnormally , Need to manually switch to normal mode ;

Imported mode

Local and Importer The cluster cannot write data during the import of ,TiDB Backend Can write data ;

Hardware requirements ：CPU/MEMORY/DISK/ network card = 32c/20G/SSD/ Wan Zhao nic
It is recommended to deploy separately

Deploy ： tiup install lightning perhaps tidb-toolkit

TiDB Lightning Applicable scenario

Support the import of mass data , 400G-500G/Hour Quick import of ,50G/Hour Transactional import of ;

Support format file ：SQL、CSV、Amazon Aurora Parquet