Redis big key problem

1 What is big key
Redis There are several common data structures in , Each structure has a large key The definition of is different , such as ：
value yes String Type ,size exceed 10KB For big key;
value yes ZSET、Hash、List、Set Is equal to the collection type , It has more members than 1 Ten thousand is big key.
2 Big key What's the impact?
We all know ,Redis A typical feature of is ： The core worker thread is a single thread .
The processing of request tasks in a single thread is serial ： If the front is not finished , I can't handle it later . At the same time, it also leads to memory data and data in distributed architecture CPU The imbalance of .

• Large execution key The client of the command itself , Time consuming has increased significantly , Even overtime ;
• Large execution key Related read or delete operations , It will seriously occupy bandwidth and CPU, Affect other clients ;
• Big key Its own storage leads to the imbalance of piecewise data in distributed system ,CPU The utilization rate is also uneven ;
• Big key Sometimes it's hot key, Frequent read operations , The impact will be great ;
• Large execution key deleted , In low version Redis Threads may be blocked in .
  So it looks big key The impact of is still obvious , The most typical is blocking threads , Concurrency is down , Cause client timeout , The business success rate of the service side decreased .

3. Big key How did it come about
Big key It is often caused by the unreasonable design of the business party , Not foreseen vaule The dynamic growth of ：
（1） Keep going value Plug data , There is no deletion mechanism , It will explode sooner or later ;
（2） The data is not segmented reasonably , Will be big key Become small key.

4. How to find big key
   4.1 Increase memory 、 Traffic 、 Timeout and other indicators monitoring
   Because of the big key Of value It's big , Threads may be blocked while reading . Lead to Redis whole QPS falling , And the client timeout will increase , Network bandwidth will rise . Configuring these alarms allows us to find large key The existence of .

• bigkeys command
  Use bigkeys The command parses... In a traversal manner Redis All in the instance key, And return the overall statistical information and the number one in each data type key.
• redis-rdb-tools
  Use redis-rdb-tools Offline analysis tools to scan RDB Persistent files , Although the real-time performance is slightly poor , But being completely offline has no impact on performance .
  redis-rdb-tools Yes, it is Python Compiling Redis rdb Snapshot file analysis tool , It can be rdb Snapshot file generation JSON file , Or generate reports for analysis Redis Details of the use of .
• Integrated visualization tools
  Based on some public cloud or internal structure of the company ,Redis There are usually visual pages and analysis tools , To help locate large key, Of course, the bottom layer of the page may also be based on bigkey s perhaps rdb The results of offline analysis of files .

5. How to solve the problem of big key problem

According to the big key The practical use of can be divided into two cases ： Can and cannot be deleted .

5.1 Delete big key

If you find something big key Not hot key, Can be in DB Query in the use of , Can be in Redis Delete ：
When Redis The version is greater than 4.0 when , You can use UNLINK The command safely deletes large files key, This command can be used in a non blocking manner , Gradually clean up incoming key.

Redis UNLINK Command similar DEL command , Means to delete the specified key. If specified key non-existent , The command ignores this .UNLINK Orders are different from DEL The command lies in , It is executed asynchronously , So it doesn't block .UNLINK The command is non blocking delete , Nonblocking deletion, in short , Is to put the deletion operation into another thread to process .

When Redis Version less than 4.0 when , Avoid blocking commands KEYS. It's a proposal to pass SCAN Command to perform an incremental iterative scan key, Then judge to delete .

Redis Scan The command iterates over the database keys in the database .SCAN Command is a cursor based iterator , After each call , Will return a new cursor to the user , The user needs to use this new cursor as the SCAN Cursor parameters for the command , To continue the previous iteration process .

• Compress and split key

When vaule yes string when , It's hard to split . Serialization can be used 、 The compression algorithm will key The size of is controlled within a reasonable range , But both serialization and deserialization are more time consuming ;

When value yes string, And it is still large after compression key, It needs to be split . A big key Divided into different parts , Record the of each section key, Use multiget And other operations to achieve transaction reading ;

When value yes list/set Is equal to the collection type , Slice according to the estimated data scale . Different elements are divided into different pieces after calculation .