The goal is

Databend It is a very advanced cloud native data warehouse based on object storage ¹ It can provide powerful computing analysis and storage capacity . Give Way MySQL DBA Very greedy . Today I want to put MySQL Of wubx Library from MySQL Full migration to Databend in . With tools ： dumpling²

Dumpling Introduce

Dumpling³ Yes support with SQL Text or CSV The format will be MySQL/TiDB Tools for data export . The original intention of the design is to replace Mydumper⁴, So the basic usage can refer to Mydumper, Of course, it is not completely copied in the implementation Mydumper, Therefore, existence and Mydumper Different usage . More help ：github.com/pingcap/tid…

I'm sorry TiDB Not provided dumpling A separate download of , Only large ones are provided package：pingcap.com/zh/product-… contains ：dumpling Binary package , Download and install are omitted .

Environmental statement

Now? MySQL in wubx Kuo is sysbench Generated data ,10 Tables , Each table 1000 All the data . Migration objectives ：DatabendDatabend Installation deployment reference ：databend.rs/doc/deploy

Use dumpling Back up the existing database

dumpling -uwubx -pwubxwubx -P3306 -h 192.168.2.10 --filetype csv -t 8 -o ./ -F 256M -B wubx

Command specification ：
-u mysql user name
-p mysql password
-P mysql port
-h mysql machine ip
--filetype csv Specify the use of CSV Format export （ It's very important ）
-t 8 Use 8 Process export
-o ./ Specify the storage location of the exported file
-F The size of the exported file
-B wubx Specify the exported database
After the command is run, the exported file is as follows ：

because Databend Yes MySQL The syntax of is not fully compatible , It needs to be removed here schema A definition of ：

engine=InnoDB

Secondary indexes

notes , This is in the new planner China has supported

in the light of sbtest Table processing can compare the explosive force ：

sed -i 's/ ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci//g' *.sql

The processed table structure is as follows ：

 cat wubx.sbtest1-schema.sql 
/*!40101 SET NAMES binary*/;
CREATE TABLE `sbtest1` (
  `id` int NOT NULL,
  `k` int NOT NULL DEFAULT '0',
  `c` char(120) NOT NULL DEFAULT '',
  `pad` char(60) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`)
);

Okay , Later we started to use streaming_load To load the data .

Use databend Of streaming load Load data

The core of importing data here is to use databend streaming load function , Specific reference ：databend.rs/doc/load-da… First, in the databend Created in wubx database

echo "create database wubx"|mysql -h 127.0.0.1 -P3307 -uroot

Because there are many tables , A script is used here cat im.sh

#!/bin/bash
t=''
getTable(){
 str=${1#*.}
 t=${str%%-*}
}

for i in `ls *-schema.sql`
do
        getTable $i
        echo $t
        cat $i | mysql -h127.0.0.1 -P3307 -uroot wubx
        time ls *$t*.csv |xargs -P 8 -I{} curl -H "insert_sql:insert into wubx.$t format CSV" -H "skip_header:1" -F "[email protected]{}" -XPUT http://root:@localhost:8000/v1/streaming_load
done

The script is rough , It can be adjusted according to the actual situation . Run the script to complete the data loading .

‍

Carry out analysis and calculation

Here the data has been loaded into Databend It's in , If you use real data , You can run the statistical analysis data test . By the way , If you want to experience databend Of join function . Can put the new planner open

set enable_planner_v2=1;

utilize show settings confirm enable_planner =1 You can use Databend Of join function . at present join almost 1-2 Weeks can be open to the public by default , If you are underused, you can also use github Give feedback on .

About Databend

Databend It's an open source 、 elastic 、 Low cost , Object based storage is a new type of data warehouse that can also be used for real-time analysis . Looking forward to your attention , Explore cloud native data warehouse solutions together , Create a new generation of open source Data Cloud.