MySQL toolset: the official export tool mysqlpump

brief introduction

Backup recovery yes DBA Core topics that cannot be bypassed , There are also many open source backup and recovery schemes in the market , But officially mysqldump Always the one at the bottom of the chain of disdain . finally , The official in the MySQL5.7 Then a new backup tool was added ：mysqlpump.

mysqlpump yes mysqldump A derivative of , I also refer to mydumper The idea of , Supports parallel data export , Therefore, the efficiency ratio of exporting data is mysqldump It will be much higher .

Introduction

mysqlpump Most of the parameters of are related to mysqldump It's the same , Overall usage and mysqldump There's not much difference . Here are some of the mysqlpump Important and commonly used parameters in .

Parameters	explain
--default-parallelism=#	Set the concurrency of parallel export , And single-transaction Conflict
--single-transaction	Create a single transaction to export all the tables
--exclude-databases=name	Exclude some libraries from export , Multiple libraries are separated by commas
--exclude-tables=name	Exclude some tables when exporting , Multiple tables are separated by commas
--include-databases=name	Include some libraries when exporting , Multiple libraries are separated by commas
--include-tables=name	Include some tables when exporting , Multiple tables are separated by commas

Practical experience

Here to mysqlpump Do a simple trial , Target instance selection MySQL 5.7, The parameter also uses single-transaction and default-parallelism, Try the effect of this conflict .

mysqlpump The output of the side refers to the following information ：

[email protected]:~# mysqlpump -h172.100.10.10 -uroot -p --single-transaction --default-parallelism=16 --set-gtid-purged=OFF -B sbtest > sbtest.sql
Dump progress: 0/1 tables, 250/987400 rows
Dump progress: 0/5 tables, 117250/3946600 rows
Dump progress: 1/5 tables, 258750/3946600 rows
Dump progress: 1/5 tables, 385500/3946600 rows
Dump progress: 1/5 tables, 516750/3946600 rows
Dump progress: 1/5 tables, 639250/3946600 rows
Dump progress: 1/5 tables, 757000/3946600 rows
Dump progress: 1/5 tables, 885000/3946600 rows
Dump progress: 1/5 tables, 1005750/3946600 rows
Dump progress: 1/5 tables, 1114250/3946600 rows
Dump progress: 1/5 tables, 1223250/3946600 rows
Dump progress: 2/5 tables, 1312500/3946600 rows
Dump progress: 2/5 tables, 1430750/3946600 rows
Dump progress: 2/5 tables, 1553000/3946600 rows
Dump progress: 2/5 tables, 1680250/3946600 rows
Dump progress: 2/5 tables, 1809500/3946600 rows
Dump progress: 2/5 tables, 1940750/3946600 rows
Dump progress: 2/5 tables, 2060000/3946600 rows
Dump progress: 2/5 tables, 2175250/3946600 rows
Dump progress: 2/5 tables, 2295250/3946600 rows
Dump progress: 3/5 tables, 2413500/3946600 rows
Dump progress: 3/5 tables, 2554500/3946600 rows
Dump progress: 3/5 tables, 2693500/3946600 rows
Dump progress: 3/5 tables, 2818750/3946600 rows
Dump progress: 3/5 tables, 2941500/3946600 rows
Dump progress: 4/5 tables, 3056000/3946600 rows
Dump progress: 4/5 tables, 3172750/3946600 rows
Dump progress: 4/5 tables, 3280000/3946600 rows
Dump progress: 4/5 tables, 3372000/3946600 rows
Dump progress: 4/5 tables, 3444750/3946600 rows
Dump completed in 126555 milliseconds

You can see that when these two parameters are enabled at the same time ,mysqlpump In fact, they are exporting tables one by one .single-transaction The priority of will be higher than default-parallelism.

Get rid of single-transaction When the test is carried out again , You will find an interesting phenomenon , Observe MySQL Of processlist, There will be the following results ：

mysql> show processlist;
+---------+------+--------------------+------+---------+------+-------------------+----------------------------------------------------+
| Id      | User | Host               | db   | Command | Time | State             | Info                                               |
+---------+------+--------------------+------+---------+------+-------------------+----------------------------------------------------+
| 2763496 | root | 172.100.10.10:49086 | NULL | Query   |    0 | starting          | show processlist                                   |
| 2763585 | root | 172.100.10.10:49192 | NULL | Sleep   |  126 |                   | NULL                                               |
| 2763586 | root | 172.100.10.10:49194 | NULL | Sleep   |  126 |                   | NULL                                               |
| 2763587 | root |172.100.10.10:49196 | NULL | Sleep   |  126 |                   | NULL                                               |
| 2763588 | root | 172.100.10.10:49198 | NULL | Sleep   |  126 |                   | NULL                                               |
| 2763589 | root | 172.100.10.10:49200 | NULL | Sleep   |  126 |                   | NULL                                               |
| 2763590 | root | 172.100.10.10:49202 | NULL | Sleep   |  126 |                   | NULL                                               |
| 2763591 | root | 172.100.10.10:49204 | NULL | Sleep   |  126 |                   | NULL                                               |
| 2763592 | root | 172.100.10.10:49206 | NULL | Sleep   |  126 |                   | NULL                                               |
| 2763593 | root | 172.100.10.10:49208 | NULL | Sleep   |  126 |                   | NULL                                               |
| 2763594 | root | 172.100.10.10:49210 | NULL | Sleep   |  126 |                   | NULL                                               |
| 2763595 | root | 172.100.10.10:49212 | NULL | Query   |  125 | Sending to client | SELECT `id`,`k`,`c`,`pad`  FROM `sbtest`.`sbtest5` |
| 2763596 | root | 172.100.10.10:49214 | NULL | Query   |  125 | Sending to client | SELECT `id`,`k`,`c`,`pad`  FROM `sbtest`.`sbtest4` |
| 2763597 | root | 172.100.10.10:49216 | NULL | Query   |  125 | Sending to client | SELECT `id`,`k`,`c`,`pad`  FROM `sbtest`.`sbtest3` |
| 2763598 | root | 172.100.10.10:49218 | NULL | Query   |  125 | Sending to client | SELECT `id`,`k`,`c`,`pad`  FROM `sbtest`.`sbtest2` |
| 2763599 | root | 172.100.10.10:49220 | NULL | Query   |  125 | Sending to client | SELECT `id`,`k`,`c`,`pad`  FROM `sbtest`.`sbtest1` |
| 2763600 | root | 172.100.10.10:49222 | NULL | Sleep   |  125 |                   | NULL                                               |
| 2763601 | root | 172.100.10.10:49224 | NULL | Sleep   |  125 |                   | NULL                                               |
+---------+------+--------------------+------+---------+------+-------------------+----------------------------------------------------+
18 rows in set (0.00 sec)

mysql>

It's obvious that ,mysqlpump Of “ Parallel export ” In fact, it is just a parallel export based on the table level , When there is a single large table , The export time will be seriously affected , There is a short board effect .

Additional questions ： If default-parallelism and single-transaction If there is a conflict , Is it impossible to confirm the data consistency when exporting in parallel ？

Practice makes truth , open general_log Take a look at the export operation ：

2021-05-12T11:54:09.033215Z        75 Connect   [email protected] on  using SSL/TLS
2021-05-12T11:54:09.075347Z        75 Query     FLUSH TABLES WITH READ LOCK // Start locking table 
2021-05-12T11:54:09.103132Z        75 Query     SHOW WARNINGS
2021-05-12T11:54:09.106382Z        75 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
2021-05-12T11:54:09.106553Z        75 Query     SHOW WARNINGS
2021-05-12T11:54:09.106640Z        75 Query     START TRANSACTION WITH CONSISTENT SNAPSHOT
2021-05-12T11:54:09.108115Z        75 Query     SHOW WARNINGS
2021-05-12T11:54:09.127277Z        76 Connect   [email protected] on  using SSL/TLS
2021-05-12T11:54:09.127452Z        76 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
2021-05-12T11:54:09.127590Z        76 Query     SHOW WARNINGS
2021-05-12T11:54:09.127680Z        76 Query     START TRANSACTION WITH CONSISTENT SNAPSHOT
2021-05-12T11:54:09.127790Z        76 Query     SHOW WARNINGS
......
2021-05-12T11:54:10.018813Z        90 Connect   [email protected] on  using SSL/TLS
2021-05-12T11:54:10.018944Z        90 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
2021-05-12T11:54:10.019047Z        90 Query     SHOW WARNINGS
2021-05-12T11:54:10.019150Z        90 Query     START TRANSACTION WITH CONSISTENT SNAPSHOT
2021-05-12T11:54:10.019226Z        90 Query     SHOW WARNINGS
2021-05-12T11:54:10.025833Z        91 Connect   [email protected] on  using SSL/TLS
2021-05-12T11:54:10.025934Z        91 Query     SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
2021-05-12T11:54:10.026048Z        91 Query     SHOW WARNINGS
2021-05-12T11:54:10.026141Z        91 Query     START TRANSACTION WITH CONSISTENT SNAPSHOT
2021-05-12T11:54:10.026219Z        91 Query     SHOW WARNINGS
2021-05-12T11:54:10.026293Z        75 Query     UNLOCK TABLES  // End lock table 
2021-05-12T11:54:10.026406Z        75 Query     SHOW WARNINGS

You can see that before parallel export , There is a thread with a global read lock , Then wait for all concurrent threads to open the transaction before unlocking the table , Therefore, the data is consistent when exporting in parallel .

Advantages and disadvantages

• advantage ：
  • Backup the database and objects in the database in parallel , Than mysqldump More efficient .
  • Better control over databases and database objects （ surface , stored procedure , User account ） Backup of .
  • Visualization of backup progress .
• shortcoming ：
  • Can only be parallelized to the table level , If a table has a large amount of data, there will be a very serious short board effect .
  • The exported data is saved in a file , The import is still single threaded , Low efficiency .
  • Unable to get the... Corresponding to the current backup binlog Location .

To sum up

Even though mysqlpump There are still many deficiencies , But compared with the original mysqldump Great progress has been made , It can also be seen from the release of this tool Oracle Finally began to pay attention to MySQL Ecological tools , Expect the government to provide more and better ecological tools .