[MySQL] online DDL details

Preface

One day , An Australian classmate I haven't seen for a long time , When we meet, we don't say hello , Ask me directly in person
Australian students asked me ： Xiao Tao Xiao Tao , My watch has 1T The data of , At this time, I want to add a column to this table , What if it can not affect the business to the greatest extent ？
At the moment , I was lost in thought , Hurry to learn

One 、 classification

stay MySQL5.6 After the version , For parts alter table, Add a new execution algorithm , Can be done DDL when ,“ parallel ” Existing business （DML operation ）.
Can pass aLter table add ALGORITHM Parameter control uses algorithms .

# For example, this way of writing 
alter  table  t1 add a int ALGORITHM=?;

At present, there are three mainstream algorithms that can be supported ：
COPY —— MySQL 5.6 Not before Online, They all execute this algorithm
INPLACE —— MySQL 5.6 The emergence of
INSTANT —— MySQL 8.0.12 The emergence of

Then there is a question , So many algorithm heads are big

Don't worry. , Xiao Tao, let's answer these algorithms easily ：
We know , It takes three steps to put Australian students in the refrigerator ： open 、 Put in 、 close .

and DDL operation , When executed , No matter what algorithm , Will go through three stages ： Preparation stage 【prepare】、 Execution phase 【DDL】、 Submission phase 【commit】. The difference is that different treatments are made in the three stages , Now let's have a good talk .

Two 、Copy

You can understand every second you hear a noun , Pure copy Ninja .

 A term is used to explain ： finger DDL when , Will generate （ temporary ） New table , Copy the data of the original table row by row to the new table , During this period, it will block DML,offline( Offline rather than online )
 For example, execute this statement ：
alter table t1 add age a int;

 Three stages ： Get ready 、 perform 、 Submit .
 Get ready ： 
1、 Data sharing lock for tables , Read frm Metadata （ here DDL Not in parallel ,DML It can be done in parallel ）
2、 Upgrade shared lock to exclusive lock ;（ here DDL、DML Can not be parallel ）
3、 stay Server Layers pass through create like sentence , Create a temporary table ,Engine Layers also generate corresponding ibd、frm file （8.0 After that, I didn't .frm file ）

 perform ：
1、 Modify temporary table metadata ( Galie )
2、 Copy the original table data to the temporary table 【 Most time consuming ,1T Data line by line copy】
3、 Delete the original table and file 
4、 Re order temporary tables and documents 

 Submit ：
1、 Commit transaction , Release the lock 

Australian students ： That's easy to understand , But now it's not practical , Because during the whole process , Are locked , Data cannot be written at this time , You can't query data , It is easy to cause database congestion
Humble waves ： Mm-hmm , How wise
Australian students ： Then how Oline DDL ah , Are you fooling me
Humble waves ： Don't worry. , Isn't it easy before difficult , The next algorithm is introduced below

3、 ... and 、Inplace

stay MySQL5.6 New algorithms appear , There is no need to copy the full table data to the new table , But it may still need IN-PLACE The way （ In situ , No need to generate new temporary tables ） Rebuild the whole table . In this case , stay DDL The initial preparation and the final conclusion of the two stages usually need to be exclusive MDL lock （metadata lock, Metadata lock ）, In addition ,DDL There will be no blockage DML.

 Get ready ：
1、 Upgrade locks for table data sharing , And upgrade to exclusive lock ;（ here DML Not in parallel ）
2、 use inplace Algorithm 
3、 The judgment statement is “rebuild table”  still  “no-rebuild”,rebuild  Create... Under the path of the original table .frm and .ibd Temporary transfer documents ;
【 Clone at the engine level , Not like it copy like that , stay server Layer creation （create like）】

 Be careful ： The operation of adding columns requires rebuild table Of 
【rebuild table】--- Generate transit file table 


++++
no-rebuild The situation of ：
 In addition to creating secondary indexes, only .frm file , Among them, the operation of adding secondary index is the most special , The operation belongs to no-rebuild Will not generate .ibd,
 But actually .ibd The file has been modified , This operation will be in the parameter tmpdir Generate a temporary file under the specified path , Used to store index sorting results , And then merge it into .ibd In file 
++++
4、 apply row log Space , To hold DDL Generated during the execution phase DML operation .（no-rebuild Unwanted ）【 stay innodb_sort_buffer In block 】


 perform ：	（online）
1、 Release the lock , Keep metadata sharing upgrade lock ;（ here DML It can be done in parallel ）
2、 Scan all data pages of the primary key and secondary index of the original table , Generate  B+  Trees , Store in a temporary file ;【 Scan in the engine layer , Most time consuming 】
3、 All changes to the original table DML The operation is recorded in the log file row log in , And playback part row_log.

 Submission phase ：
1、 Upgrade metadata sharing upgrade lock , Generate exclusive lock table ;（ here DML Can't be parallel ）
2、 redo row log The content in ;（no-rebuild Unwanted ）
3、 Rename the original table file , Rename the temporary file to the original table file name , Delete the original table file ;
4、 Commit transaction , Change complete .

Be careful ：

• stay DDL Data generated during , It will operate as normal , Write to the original table , remember redolog、undolog、binlog, And synchronize to the slave library to execute , It's just that the extra will be recorded in row log in , And write row log The operation itself will also be recorded redolog
• And only in the submission stage row log redo , This stage will lock the table , At this point, the main library （ New table space +row log） And from the library （ Table space ） The data are consistent , In the main warehouse DDL Submit and execute , This DDL Will write binlog Transfer to and from the library to execute , Execute the... From the library DDL when , This DDL For the local library, it is still online Of , That is, writing data directly from the local database will not block , It will also be generated like the main library row log.
• But for the main database synchronization DML, It will be blocked , yes offline Of ,DDL It's an exclusive lock, and it's the same in the copy thread , So it will not only block the table , But all subsequent operations synchronized from the main database （ It is mainly exclusive when the replication thread is parallel , At the same time, only himself is performing ）. So big watch DDL operation , It will cause synchronization delay .

here , Australian students Another question ： Pretty good , Then why is Online Well ？
Humble waves ： Although there are three stages （ Get ready 、 perform 、 Submit ） in , There are two stages （ Get ready 、 Submit ） Can't do it CRUD, But actually , Whole DDL The implementation phase takes the longest proportion , for example 30 Minutes of DDL, Get ready + The submission phase only occupies 1 Minutes of time , The rest 29 Every minute , So for the business layer , Most of the time, it can be accessed normally , So I did Oline DDL 了 .
Australian students ： EH , Don't you want to 1 Minutes will jam , I just want to insert a large amount of data in that minute , You are not familiar with this melon 🤪

Humble waves ： Line line , Now let's introduce the next algorithm

Four 、Instant

Just modify the metadata in the data dictionary , There is no need to copy data or rebuild the whole table , Again , There is no need to exclude others MDL lock , The original table data will not be affected . Whole DDL The process is almost instantaneous , It won't block DML.

This new feature is 8.0.12 Introduced （ tencent DBA Team contribution ）. perform DDL In operation ,ALGORITHM Options can be left unspecified , Now MySQL according to INSTANT、INPLACE、COPY Automatically select the appropriate mode in the order of . You can also specify ALGORITHM=DEFAULT, It's the same effect . If you specify ALGORITHM Options , But if you don't support it , Will report an error directly .

When COPY Mode time , At this time, any data modification operation in the table ,DDL Will be blocked .COPY A temporary new table will be generated in mode , After the operation, the original table will be deleted , The new table is renamed to the original table name . When DDL After start , The original table can only be read-only , Other DML Operations will also be blocked .COPY In the process , The only time to block read-only is when cleaning up the old table structure and table definition cache .

MySQL 8.0.12 A new algorithm just put forward , At present, only a few operations such as adding columns are supported （ It's not mature yet , Enterprises are generally 5.6、5.7 edition ）, utilize 8.0 New table structure design , You can modify the table directly metadata data , Forget it rebuild The process of , It's greatly shortened DDL Statement execution time .

5、 ... and 、 Some additions

Australian students ： That support Inplace Algorithm DDL It must be Online Yes, I don't know ？
1、 Conceptually speaking ,INPLACE and Online It's a matter of two different dimensions ;
2、Ccopy and Inplace refer to DDL Internal execution algorithm , Can be interpreted as ：COPY Is in Server layer （ Service layer ） The operation of ,INPLACE Is in InnoDB layer （ Storage layer ） The operation of ;
3、 Actually , For users , Care for Online Whether or not , Usually only related to one problem ： Whether concurrency is allowed DML;
4、 In a classic philosophical term ：Copy The algorithm performs DDL Definitely not Online Of ,Inplace The algorithm performs DDL Is not necessarily Online Of .

6、 ... and 、 summary

COPY —— 5.6 I used to adapt to this method by default , The whole process cannot be parallel DML
INPLACE —— MySQL 5.6 The emergence of , During the short rental period starting and submitting , Not in parallel DML
INSTANT —— MySQL 8.0.12 The emergence of , The only time to block read-only is when cleaning up the old table structure and table definition cache .

The above is purely program effect , Classmate Ao is my best friend , Graduated together
This time, , I really graduated , Four years of College passed quickly
＂ therefore , Don't look back , Walk away .
I will probably miss the mottled shadow of the trees in the mall ,
Miss the setting sun on the South basketball court ,
Miss the lights at night ,
Umbrella in the rain ,
Morning fog ,
I probably miss every smile and every tear .
I wanted to harvest a spring breeze , But you gave me the whole spring ;
I wanted to pick a red leaf , But you gave me the whole maple forest ＂​
I am motivated to be an excellent DBA My classmate Xiao Tao , Those who have questions are welcome to discuss in the comment area , Let's work together , To rush ！ See you in the next blog post ~