Online data migration scheme encountered in the project 1 - general idea sorting and technical sorting

The project has a requirement , Split the old library into the new one , Migrate the old data to the new database , During this period, additional recording and migration of data should be considered , Because no canal Sync , The task I am assigned is to add an interface for adding migration data to the code to ensure smooth migration and data consistency

Because there is a large amount of online data and data will be entered into the table all the time , A few points to consider ：

1. First, sort out the table of continuous growth data during the wee hours , Tables stored in normal time are generally not written in the wee hours , Or limit it in advance and tell the user not to enter the table during the wee hours , Therefore, it can be ignored not to migrate
2. dba transfer 1 All data before clicking , Suppose the migration time is one hour ,2 End of point migration ,2 Double write after clicking , Data in both old and new data sources will be written
3. Configure multiple data sources , Realize user-defined annotation to switch data sources
4. I will write an open interface , Pass in the specified time range ( If 1 spot -2 spot ), There will be deviation between new and old data sources , I just need to pass in the time range parameter , Interface migration 1-2 Data between points
5. This interface does two things , Add and modify , Pull... Separately create_time and update_time All data in this time period , Here must be submitted in advance DB To apply for a work order, index these two fields online
6. First check from the old table create_time Range data , Because it is double written , So the entity classes found in the old data source list Need not pass ID Insert new data source （ And insert in pages ） Ensure that the migrated data is consistent , however ID No guarantee （ Because it's self increasing ID There may be problems ）
7. Check the old table again update_time Range data , also sql much update_time != create_time To ensure that only updated old data is found , When migrating here ID Is to ensure consistency , According to the legend ID updated （ You can use the index here , Currently online log The data volume of the table has ten million levels , But there is not much data in an hour in the morning , Batch update It is not expected to be too slow , The efficiency needs to be verified later ）