MySQL DDL change

Preface         

As the business grows , Users have more and more demands on the system , This requires the system to be able to quickly update iterations to meet business requirements , Usually when the system version is released , You have to execute the database first DDL change , Including creating tables 、 Add fields 、 Add index 、 Modify field properties, etc .

In the case of a large amount of data , perform DDL All very soon , It has little impact on the business , But when there's a lot of data , And our business has done the separation of reading and writing , Access to the real-time data warehouse , At this time DDL Change is a problem , We need to consider the business of all parties .

Now let's talk about our company in these years ,MySQL Medium DDL Changes in the way they are executed 、 The big table DDL How to choose and DDL Execution process monitoring .

MySQL Medium DDL

DDL summary  (Data Definition Language) 

MySQL Medium DDL There are many sentence forms , To sum up, there are the following categories ：CREATE,ALTER,DROP,RENAME,TRUNCATE.

These operations are implicitly committed and atomic , Or success , Or failure , stay MySQL 8.0 Before DDL Operations are not logged .

Today, let's talk about the database structure changes related to the system release , Mainly ALTER TABLE Changed ,DDL The change process is normal DML Changes are similar , As shown below

notes ： This involves MySQL Basic knowledge of , If you don't know, look at me MySQL The basic chapter is enough .

Early MySQL edition ,DDL Changes will cause the entire table to be locked , Block the table DML operation , Affect the normal operation of the business , The good thing is , With MySQL Iteration of version ,DDL And the way we do it is changing .

MetaData Metadata

MySQL Metadata （MetaData） With the others RDBMS The database is the same , Structural information of the described object , Stored in information_schema Under the architecture , For example, common TABLES、COLUMNS etc. , The following example is to create a table crm_users,MySQL Will automatically go to Information_schema.tables and columns Insert data into the relevant data dictionary table , This data is called metadata , It's usually static , Only on the watch DDL The operation will update in real time .

MetaData Lock

MySQL utilize MetaData Lock To manage access to objects , Ensure data consistency , For some core business tables , On the table DML The operation is frequent , Adding fields at this point may trigger MetaData Lock.

You can see Waiting for table metadata lock Events wait ,thread 155 Being implemented alter table wait for thread 154 Executive select Release the lock , because DML Will hold during execution SHARED_READ lock , To execute DDL When you get SHARED_UPGRADABLE（ Shared upgradeable locks , Abbreviation for SU, Allow concurrent updating and reading of the same table ） Lock successful , But get EXCLUSIVE MetaData Lock Lock failed , On hold PENDING state .

DDL Way of execution

from MySQL Official documents can be found ,ALTER TABLE A lot of options , The main performance related options are ALGORITHM and LOCK.

ALGORITHM OPTION	DESCRIPTION
COPY	MySQL Early ways to change , You need to create a modified temporary table , Then copy the data of the original table to the temporary table according to the data row , do rename Rename to create , Concurrency is not allowed during this period DML operation , The original table is readable , Don't write , At the same time, it needs twice as much disk space .
INPLACE	Modify directly on the original table , No need to create temporary tables, copy data and rename , The original watch will hold Exclusive Metadata  Lock, Usually, concurrency is allowed DML operation .
INSTANT	MySQL 5.8 Start supporting , Only modify the metadata in the data dictionary , Table data is not affected , There was no during the execution Exclusive Metadata  Lock, Allow concurrent DML operation .

You can see from this list that ,MySQL about DDL The execution mode has been optimized , The purpose is to improve DDL Execution efficiency , Reduce lock wait , Does not affect table data , At the same time, it does not affect the normal DML operation .

LOCK Options

LOCK OPTiON	DESCRIPTION
DEFAULT	The default mode ：MySQL According to the operation , Select automatically without locking the table as much as possible LOCK Pattern .
NONE	unlocked ： allow Online DDL Concurrent read and write operations are performed during , If Online DDL Operations do not support concurrency on tables DML operation , be DDL operation failed , Invalid for table modification .
SHARED	Shared lock ：Online DDL The read is not affected during the operation , Blocking write .
EXCLUSIVE	Exclusive lock ：Online DDL No operation is allowed on the lock table during operation .

The following examples illustrate the execution process of these methods , First create the test table , Make some data .

COPY

COPY The change process is as follows ：

According to business needs , Need to be in crm_users Add a field user_type, use COPY How to implement the change .

From the execution process and profile It can be seen that , adopt COPY Method creates a temporary table #sql-564_85, obtain System Lock, Copy data to temporary tables , Do it last rename Table name switch , Release Lock resources , Concurrency is not supported during execution DML operation .

INPLACE

INPLACE The way is to modify the original table directly , For adding indexes 、 add to / Delete column 、 Modify fields NULL/NOT NULL Properties and other operations , Need modification MySQL Internal data records , Need to rebuild table （Rebuild Table）.

We can see from the execution process that , Need to get Exclusive Metadata  Lock, Modify table data , Release Lock, Support concurrency during execution DML operation .

INSTANT

MySQL 5.8 The way to start rolling out ,DDL Only modify the metadata in the data dictionary , Table data is not affected , No, Exclusive Metadata  Lock, Allow concurrent DML operation , Supported by DDL There are limits to change , At present, it mainly includes adding fields , add to / Delete generated columns , modify ENUM or SET Column , Change the index type and rename the table .

Compare the execution efficiency of these three methods

Way of execution / project	Data volume (w)	execution time (s)	Rebuild table	modify MetaData	modify Data	Allow concurrency DML
COPY	650	29.89	YES	No	Yes	No
INPLACE	650	10.56	YES	No	Yes	Yes
INSTANT	650	0.19	No	Yes	No	Yes

ONLINE DDL

end MySQL 8.0,OnLine DDL There are three ways COPY,INPLACE,INSTANT,MySQL It's automatically executed according to DDL Choose which way to use , Generally, priority will be given to INSTANT The way , If not , Just choose INPLANCE The way , If you don't support it, you have to choose COPY Way .

MySQL Official documents also show Online DDL Support matrix of , List the commonly used DDL operation , The contrast items mainly include whether to rebuild the table , Allow concurrent DML Operation and only modify metadata , Table data is not affected .

Operation	Instant	In Place	Copy	Rebuilds Table	Permits Concurrent DML	Only Modifies Metadata
Adding a column	Yes	Yes*	Yes	No*	Yes*	Yes
Dropping a column	No	Yes	Yes	Yes	Yes	No
Renaming a column	No	Yes	Yes	No	Yes	Yes
Setting a column default value	Yes	Yes	Yes	No	Yes	Yes
Dropping the column default value	Yes	Yes	Yes	No	Yes	Yes
Changing the auto-increment value	No	Yes	Yes	No	Yes	No
Making a column NULL	No	Yes	Yes	Yes*	Yes	No
Making a column NOT NULL	No	Yes	Yes	Yes*	Yes	No
Adding a primary key	No	Yes*	Yes	Yes*	Yes	No
Dropping a primary key	No	No	Yes	Yes	No	No
Creating or adding a secondary index	No	Yes	Yes	No	Yes	No
Dropping an index	No	Yes	Yes	No	Yes	Yes
Renaming an index	No	Yes	Yes	No	No	No
Adding a FULLTEXT index	No	Yes*	Yes	No*	No	No

The big table DDL programme

In the actual business system , Business development is relatively fast , The table has a large amount of data , At the business level, there is a separation between reading and writing , At the same time MySQL Data is synchronized to the data warehouse in real time （ Including real-time data warehouse and offline data warehouse ）, The actual database architecture is as follows .

Suppose this is a trading system database , The order sheet booking Yes 8000w data , And access to real-time and offline warehouses , According to business needs , In the order form booking Add a field , stay MySQL 5.7 Adding fields before is a high-risk operation , The impact on the business needs to be fully considered , It mainly exists in two aspects ：

1. In the read-write separation scenario , The delay of master-slave synchronization results in inconsistent business data

2. Real time data warehouse ADB Source side is not allowed MySQL Table rename , If you pass COPY Way or pt-osc、gh-ost Tools like that rename Table name , Then you need to delete the table from the database , Reconfigure synchronization （ Total quantity + The incremental ）, It will affect the warehouse business

ONLINE DDL The way

about MySQL 5.6 To 5.7 Version of , have access to OnLine DDL Changes in the way , For a big watch , It will take a long time to execute , The advantage is Master On DML Operation is not affected , But it will lead to master-slave delay .

If Master Add a field on the 20 minute , Corresponding Slave We should also carry out 20 minute , in the meantime Slave It's been delayed , It will cause inconsistent business data , Let's say the user Master checkout success , because Slave Delayed query order information , The reason why the order was placed by the user was not wrong , Another order was placed , A situation that leads to repeated orders .

In this way, the master-slave is delayed , But it will not affect the business of real-time data warehouse , According to the business situation , You can only choose to execute in the low peak period of business .

pt-osc Tools

In order to solve DDL The impact of the master-slave delay caused by the change on the business , You'll think of using a big watch to change your weapon pt-osc（pt-online-schema-change） perhaps gh-ost Tools to do , The implementation process and principle of the two tools are similar , The change process is as follows （ Don't think about foreign keys , according to MySQL The specification does not allow the use of foreign keys ）：

1. Create a new table , The table structure is the modified data table , Used to import data from a source data table into a new table .

2. Create triggers on the source table , Used to record data from the beginning of copying , Continue the data modification operation on the source data table and record it , Used after data copy , Perform these actions , Ensure that data is not lost .

3. Copy the data , Copy data from the source data table to the new table .

4. Modify the foreign key related sub table , According to the revised data , Modify the child table associated with the foreign key .

5. rename The source data table is old surface , Put the new watch rename Is the source table name , And will old Table delete .

6. Delete trigger .

perform pt-osc You also need to get one Exclusive Metadata  Lock, If during this period there is DML The operation is in progress ,pt-osc The operation will always be suspended PENDING state , It's normal at this time DML Operation will be blocked ,MySQL The number of active connections skyrocketed ,CPU Usage rate 100%, The dependent interface of this table will report an error , So we should choose to execute in the low peak period of business , At the same time MetaData Lock Lock monitoring so that business is not affected , Let's look at an example ：

D=trade, t=booking： database trade, Table name booking.

--chunk-size=1000： The number of data rows per copy .

--max-log = 1： Make sure the slave delay does not exceed 1s, Stop copying data if you exceed it .

--check-interval=2： Said wait 2s And then continue to copy the data .

--recursion-method="hosts"： If you don't use the default port 3306, So use hosts It's more reliable to find the slave Library in the way of .

commonly MySQL binlog The format is ROW,pt-osc In the process of copying data, a lot of binlog, It may also lead to master-slave delay , You need to control the size and frequency of each copy , During execution , It will also lower DML The concurrency of .

MySQL 8.0 Change the way

Used to Oracle Of all know ,DDL Changes are all about modifying metadata , Hundreds of millions of watches in Oracle in DDL Changes are all done in an instant .

What's exciting is ,MySQL 8.0 Also launched INSTANT The way , Really just modify MetaData, Does not affect table data , So its efficiency has little to do with table size . It is suggested that the new system be put on line MySQL Try to use MySQL 8.0, Old databases can also be upgraded to MySQL 8.0 Get better performance .

The official document is right INSTANT The explanation of ：

INSTANT: Operations only modify metadata in the data dictionary. No exclusive metadata locks are taken on the table during preparation and execution, and table data is unaffected, making operations instantaneous. Concurrent DML is permitted. (Introduced in MySQL 8.0.12)

Both master and slave synchronization should be solved , And to solve rename The problem of asynchronous data warehouse , At present, only INSTANT Way to meet the needs .

monitor DDL Implementation progress

Execute in the big table DDL When changing , Very concerned about the progress of its implementation ,MySQL 5.7 There was no good tool to monitor before , Basically, I have to wait . stay MySQL 8.0 You can turn it on performance_schema, open events_stages_current Events are monitored .

summary

DDL The process of iterating in the business system version is essential , How to... Without affecting the business and peripheral systems , Realization DDL The smooth change of , It is necessary to consider the characteristics of the system , Assess the importance and priorities , At the same time, we should master the difference MySQL edition DDL Way of execution , So that we can make better choices .

For example, it is mentioned above that , Now I'm on the big data team , Our business has done the separation of reading and writing , Access to real-time data warehouse at the same time , Shucang doesn't support rename operation , At this time, you can choose to use in the low peak period of business ONLINE DDL By , Minimal impact on business systems , At the same time, it doesn't affect the warehouse .