Troubleshooting of soaring memory utilization of redis cluster instances

On a sunny afternoon , Suddenly, a cache instance in the production environment issued a memory utilization exceeding 90% Alarm of , Then I will immediately see what the situation is with my friends .

The phenomenon is like this , The memory utilization of an instance in the cluster exceeds 90%, The slave node of this instance , Memory usage is low . And the memory utilization of other partitions is very low , Only this slice is high . See the picture below cachecloud Instance state diagram .

The first thought must be big key The cluster is tilted , So we first put this piece of data dump come out , And then use rdbtools Tool analysis dump file , Put the big key To find out , Then contact the business personnel .

Doing it rdb While analyzing , Also consider , Is it possible that someone has secretly connected to the instance , And then used monitor Forget to turn it off , Then, log in to the server where the instance is located and connect to the instance to use client list Command to view the connection of the client , See if there is a belt monitor Of relevant commands . Found that there was no .

~]$ redis-cli -p 9999 client list | grep monitor
~]$ 

On the other hand, use the connection instance to use bgsave Command to generate dump Data file colleagues also found that , The size of the generated data file is only 130 many M, Although the analysis also found a big key, But the size is only 130 many M, It's a possession 360 More than ten thousand elements List.

But this should not be enough to cause 9 individual G Memory usage of . I thought it was bgsave The command is not executed properly ？ I did it again bgsave command , Check the execution time and dump Data validation is indeed 130 many M. Use info Command to view instance information , I can't see the problem .

At a time when there is nothing to do , My colleague brother Liang learned from the example of client list The result of the order shows why .

]$ redis-cli -p 9012 client list | grep -v "omem=0" 
id=11164727 addr=10.4.7.48:50478 fd=458 name= age=4230 idle=303 flags=c db=0 sub=0 psub=0 multi=-1 qbuf=163072064 qbuf-free=0 obl=0 oll=659943 omem=8546641352 events=rw cmd=lrange

This filtered client link ,omem Why 8 individual G.omem What is the value . It is redis The amount of memory used by the service for the client connection output cache .

see redis Slow log of the instance , You can see the following results , identical List key Of lrange Inquire about , Parameter is 0 -1.

After thinking , It's probably understandable , The instance that uses this cache has a List Large of type key, Yes 360 More than ten thousand elements , Probably 130 many M, Then the client passes frequently lrange keyname 0 -1 To get all the elements in full . The cache server receives an instruction , Time consuming 700000 Multi microsecond （700 Many milliseconds ） Throw the corresponding data into the output buffer , Then he went on to deal with other instructions , Then because this instruction is called very frequently , The data in the output buffer will accumulate more , But the client receives data at a limited speed . The image point can be understood as a pool with water at the same time , The water discharge speed is less than the water receiving speed , The one who receives the water is Redis The result obtained by the service processing instruction , The client receives the returned results , This pool is the output buffer , It also needs to occupy Redis Instance memory , Enough time , Sooner or later the pool will be full . This is the same. Redis The reason why the instance memory keeps growing .

because Redis The amount of data in the instance does not change , So the Slave The memory of the instance has not changed .

Looking back , If master The memory of the instance has skyrocketed , however slave Instance memory and master When there is a large memory gap between instances , It's probably monitor Command not closed or output buffer backlog .

Finally, we manually connect the corresponding client kill fall , Then the memory shudders down .

~]$ redis-cli -p 9012 client list | grep -v "omem=0"
id=2253936 addr=10.4.7.48:52303 fd=276 name= age=7580 idle=36 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=1581 omem=20931976 events=rw cmd=lrange
id=2252693 addr=10.4.7.48:51712 fd=252 name= age=9268 idle=230 flags=c db=0 sub=0 psub=0 multi=-1 qbuf=79914221 qbuf-free=0 obl=0 oll=504978 omem=6682230512 events=rw cmd=lrange
id=2258097 addr=10.4.7.48:54497 fd=345 name= age=1172 idle=194 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=254068 omem=3361306536 events=rw cmd=lrange
id=2257345 addr=10.4.7.48:54063 fd=401 name= age=2438 idle=167 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=1642 omem=21713784 events=rw cmd=lrange
~]$ redis-cli -p 9012 client kill 10.4.7.48:51712
OK
~]$ redis-cli -p 9012 client kill 10.4.7.48:54497
OK