Summary of 2 billion redis data migration

Today it is 2 Zhou's migration work has been completed , Here is a summary 、 replay .

One 、 Migration background

1、 on-line 8 platform 32G Of machine memory , use allkeys-lru The elimination strategy of traditional Chinese Medicine , The main data types are string And there is no expiration date . Every machine is 2.4-3.2 100 million .
2、 I gave it to 32 Taiwan machine , The maximum content of each machine 10G, Need to put 20 Billion data points scattered to 32 On the stage .

Two 、 Migration plan

 First , Online enable double write , Synchronize new data with old data redis On the machine .
 secondly , Determine the migration plan .
      Scheme 1 、dump come out rdb File to each machine , Load recovery , It is obvious that the maximum memory limit of the new machine cannot be met dump requirement 
      Option two 、 Program migration , use scan Scan out... On each machine key, Re pass key Find the corresponding value.
  We can know from the background , Only option 2 can be selected .

3、 ... and 、 Problem consideration

1、scan Whether data will be lost 
       I'm sure ,scan No data will be lost , But it may repeat key, This is mainly caused by shrinkage , Specifically, you can ask Du Niang or take a look redis Of rehash The source code of that piece .
2、scan Is it very performance consuming 
     scan What is being scanned is what is in the dictionary key, these key All are hash On a bucket , Every time scan after , Will return the next cursor , Therefore, whether the performance is consumed or not has a considerable relationship with the number of pieces you scan each time . The number of entries to be fetched each time depends on your business and operation .
3、 What if the program crashes during execution ？
      This requires your program to record the cursors currently executing .
4、 That language implementation 
	 This migration java、php、go Fine , I use go+redigo As an implementation , Make full use of goroutine and channel.
5、 Compare the data information before and after migration 
     When performing the migration task, first put the old and new machines info Print the information into the log , The finished information should also be printed into the log for direct comparison .

Four 、 Specific ideas

1、 One scan coroutines , Every scan 200 strip , hold key from redis It's scanned from the Internet , And then throw it in read In the pipeline .
2、15 individual read coroutines , Read... In the pipeline key, adopt pipeline+dump(dump It's an order. ) Use the command to fetch the data ,
 Then group them , Throw it write In the pipeline .
3、20 A coroutine reads write Pipeline data , Pass the data through hash The algorithm is right again key Group through pipleline+restore(restore It's an order. ) To the corresponding machine .
4、8 After all the machines have been executed , Then turn on the detection keys Whether the method already exists , Supplement the data that failed to write .

5、 ... and 、 Problems often encountered in migration

1、 Every scan 200 strip ,2.4 100 million data 40 The migration can be completed in minutes , Single process cpu Utilization rate to 370%, The average bandwidth is 120Mb,
 If these two indicators cannot be accepted , You can put 200 Strip size reduction .
2、 Focus on the problems encountered 
       After migration , It is found that there are more old machines than new ones 3000 Ten thousand data , adopt exists key After testing , Still less 3000 Ten thousand , adopt down Come out with one less machine on all key Contrast , Found duplicate key, That is to say a key On two machines . The probability is before , It is caused by the direct expansion of the machine without data migration .
 3、 Data detection 
      scan On the new machine key, From the number of entries taken each time , Pick a few at random and compare them on the old machine .
 4、 Log problems 
      Because the migrated data is large , Therefore, the program debugging of log should be complete , After the actual implementation , Try not to , Otherwise, a machine will come down and use 2G The left and right logs are too big .

This completes the migration , If you need code reference, you can leave a message .