An article to solve the bigkey problem in redis

Some time ago , To see this BigKey The problem of , Because the understanding is vague and uncomfortable , So there is the following summary .

Take it all for him this time , It's clear

Here comes the main dish

redis Medium BigKey Problems and solutions

BigKey What's the problem

BigKey The concrete manifestation of is redis Medium key Corresponding value It's big , The amount of redis The space is quite large ,

Big in essence value problem

.

about redis Different data structure types in , There are several common examples :

●  about String Type of value value , Value exceeds 10MB（ The data value is too large ）;

●  about Set Type of value value , The number of members included is 10000 individual （ Large number of members ）;

●  about List Type of value value , The number of members included is 10000 individual （ Large number of members ）;

●  about Hash Format value value , Number of members included 1000 individual , But the total of all member variables value The value is 1000MB（ The total volume of the members is too large ）;

BigKey How did the problem arise

• redis Medium key-value The key value pair is not set properly , cause key Corresponding
  value value
  Particularly large .

• about list,set This type of structure ,
  Invalid data
  Not deleted in time .

• Inaccurate business analysis , Lead to
  In real business value value
  Too big , Such as hot issues .

BigKey How to locate the problem

• Use redis The order that comes with you  redis-cli --bigkeys  Online scan big key, The information displayed is not detailed , And this command is not blocked , So it doesn't matter redis Normal use of .

• Using third-party tools redis-rdb-tools, It will be used first in the process of use bgsave command dump One rdb Mirror image , Then analyze the image , because bgsave yes redis A child thread in the generates images , Does not affect redis External services .

• debug object Method

C:\Users\93676\Desktop>redis-cli
127.0.0.1:6379> set a "sadasd"
OK
127.0.0.1:6379> debug object a
Value at:00007FCD3EC12FE0 refcount:1 encoding:embstr serializedlength:7 lru:14754079 lru_seconds_idle:15

serializedlength

It means a This key Corresponding value after embstr Number of bytes after compression , There is one point involved here redis Medium

String The underlying structure

After ,redis3.2  After the version , When string When it's not a numeric string , Bytes greater than 44, Use raw How to store , When it rains 44 Bytes are used embstr How to store , That is, the compressed string .

BigKey Impact

• Memory consumption increases

Compared with normal Key, Read large key The memory required will increase , If it looks like value Medium list Growing , It may cause OOM（ out of memory ）, Another is to achieve redis Set the maximum memory maxmemory Value will cause write blocking or partially important Key By redis The elimination strategy of has been deleted .

• The network congestion delay increases

Reading big key When , Because there are many contents to be read , Occupy a large bandwidth , Cause network bandwidth congestion , It will also affect other applications , Cause the network congestion delay to become larger .

• IO The blocking delay increases

BigKey Corresponding to the problem value Relatively large , When it comes to larger value To read and write , It takes a long time , This may block subsequent request processing , In fact, it's mainly BigKey Of value Conduct IO It takes a long time to write ,IO Writing is to return the processed value to the requester through the network , because Redis The core thread of is single thread , That is, the worker thread is a single thread , Task processing in a single thread is executed serially , After the previous task is completed , Only later tasks can be continued , So because of the single BigKey The reason may cause IO Blocking delay .

• BigKey It's difficult to move

The problem is that

Redis colony

in , When you need to be big value Migration （ about Redis  colony slot Migration ）, Mainly used migrate Command to migrate ,

migrate By command dump and restore and del Three command combinations

Atomic command complete , If there is bigkey Words , Maybe because of the big value The reason for the migration failure , And slower migrate It will block Redis, influence Redis service .

BigKey How to solve the problem

• in the light of BigKey Conduct
  Split

By way of BigKey Split into multiple small Key The key/value pair , And the corresponding value The size and the number of members split into are reasonable , Then store it , Pass when obtaining get Different key Or with mget Get stored key value pairs in batch .

• clear
  Invalid
  The data of

This is mainly aimed at things like list and set This type of , In use ,list and set The corresponding content in is increasing , But because what was stored before is invalid , Need regular right list and set Clean up .

• Compress the corresponding BigKey Of value

You can serialize or compress value Compress , It becomes smaller value, But if after compression, if the corresponding value Or especially big , You need to use the split method to solve .

• monitor Redis Medium memory , bandwidth , growth rate

Through the monitoring system , monitor redis Memory usage and network bandwidth usage in , And the growth rate of memory occupation in a fixed time , When the set threshold is exceeded , Handle alarm notification .

Warm reminder ： This is a little bit more , Please look down carefully

expand

heat key problem

be relative to BigKey, heat Key It is also a point worthy of attention .

heat key

： about Redis One of the Key The received QPS、 Significantly higher than others Key, We call it heat Key, Common heat Key Such as

heat key The problems that arise

： heat Key Because of frequent visits . Take up a lot of Redis CPU Time degrades its performance and affects other requests ;

heat key Solutions for

： You can use

Read write separation architecture

, If it's hot Key Is generated from read requests , Then reading and writing separation is a good solution . When using the read-write separation architecture, you can reduce each... By continuously increasing the number of slave nodes Redis Read request pressure in the instance .

summary

about BigKey The reason for the problem ： That is to say Redis The key value pair corresponds to value The larger .

about BigKey The discovery of the problem ： You can scan and find through third-party tools or built-in commands .

about BigKey Solutions to problems ： By splitting , Compress , Clean up invalid data and other methods for these big value Just deal with it .