How to do smooth data migration: Double write scheme

1. What is data migration

Data migration refers to transferring a batch of data from a homogeneous storage system （ Such as MySQLA To MySQLB） Or heterogeneous storage systems （ Such as MySQL-MongoDB） Transportation and migration between .

The simplest way of data migration is to move data through scripts or scheduled tasks , Or through similar canal Such tools for data synchronization .

This simplest scheme can work well when the amount of data is small and the system has low requirements for data consistency , But for real-time online systems with a large amount of data , The following three guarantees need to be made in the process of data migration ：

• Online migration ： Migration should be online , That is, data will be written while migrating
• Data consistency ： Data should be complete , That is to say, after the migration, it is necessary to ensure that the data of the new library is consistent with that of the old library
• Roll back ： The migration process needs to be rolled back , In this way, once problems occur in the migration process , You can rollback to the source library immediately without affecting the availability of the system

2. Data double write migration scheme

Data double write migration is the most commonly used data migration scheme , It can ensure that the migration process is online 、 The data is consistent before and after migration 、 The migration process can be rolled back . The data double write migration scheme is divided into five steps , Namely ： Sync 、 Double write 、 check 、 Cut and read 、 Cut and write . With MySQL Data migration to MongoDB As an example to illustrate the five step process .

2.1. Sync

First, through Binlog Monitoring tools （ Such as Canal） obtain MySQL Full log and incremental log , And keep writing MongoDB：

2.2. Double write

After synchronization , here MongoDB The data in is almost the same as MySQL Agreement , But at this time, you cannot directly switch the read / write operation to MongoDB in , Because there will be data inconsistency in the process of switching , At this point, you need to double write .

The so-called double writing is actually writing in the program MySQL when , Also convert this data into format and write it at the same time MongoDB, In order not to affect the interface written pct99 Time , Can be right MongoDB Write to asynchronize , For the data that fails to be written, it will be retrieved and written regularly .

Pay attention to Binlog The listener is also writing data incrementally , Therefore, you can control the double write startup through the switch , close Binlog Monitor program .

2.3. check

After two steps of synchronization and double writing , here MySQL and MongoDB Online real-time synchronization should have been maintained , But in order to ensure that there is no problem with double write data , It is necessary to correct MySQL And MongoDB The consistency of data is checked by sampling , When the data is verified to be inconsistent , With MySQL The data is subject to coverage MongoDB Corresponding data in .

2.4. Cut and read

When sampling verification lasts for a period of time （ Like a week ） when , If the data consistency is basically 100% when , Can start read cut .

Cut reading refers to reading data , incorrect MySQL To read , It's right MongoDB To read . At this time, small flow data is read through gray control , After observing for a period of time, there is no problem , Gradually enlarge the cutting and reading flow , until 100% read MongoDB.

2.5. Cut and write

After reading , When reading, it is no longer dependent on the old MySQL, It depends on new MongoDB 了 . The last step to do at this time is to cut and write , That is, turn off double writing , Just write MongoDB. At this point, the data migration is complete , Pass both reading and writing MongoDB.