[redis problem] record a big key problem handling

background

I received an alarm today ,redis cluster Memory usage of a partition node of 85% 了 . Go on like this
It is estimated that the weekend will be blasted .

The screening process

1. First, check the monitoring , Indeed, other nodes are normal ,
But this node is highly used . Monitoring is as follows ：

2. This node has been used too much memory , My colleagues have moved once before ,
So this time it happens again , I think the problem is not simple relocation slot Can the , So further investigation .
The first thing to consider is whether it is big key The problem of .
After the big key analysis , There are indeed large key, Big key The analysis is as follows ：
Big analysis key Pay attention to the system CPU Usage rate , because SCAN It costs a lot of CPU

# The order is as follows ：
$ redis-cli -h ip --bigkeys

# And then start scanning key
# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type. You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).

[00.00%] Biggest zset   found so far 'feeds' with 200 members
[00.00%] Biggest hash   found so far 'ursor.14620821' with 2 fields
........# The intermediate process is omitted here 
[79.66%] Sampled 5000000 keys so far
[80.55%] Biggest set    found so far 'device' with 11 members
[95.59%] Sampled 6000000 keys so far

-------- summary -------

# You can see the total number of scans 600 many w Of key
Sampled 6276594 keys in the keyspace!
Total key length in bytes is 184805760 (avg len 29.44)

# Then, it will analyze the large..., which exist in several types respectively key
Biggest   list found 'followed_queue' has 2360286598 items
Biggest   hash found 'cursor' has 7 fields
Biggest string found 'user' has 62505 bytes
Biggest    set found 'device' has 11 members
Biggest   zset found 'popular' has 270505 members

1 lists with 2360286598 items (00.00% of keys, avg size 2360286598.00)
2615478 hashs with 2859295 fields (41.67% of keys, avg size 1.09)
139674 strings with 34883495 bytes (02.23% of keys, avg size 249.75)
0 streams with 0 entries (00.00% of keys, avg size 0.00)
3 sets with 15 members (00.00% of keys, avg size 5.00)

Analyze the above results ：
notice list There is one. 23 Billion yuan list, From the name, we can see that it is a queue ,
Take another look at this key Size ：

IP:6379> memory usage followed_queue
(integer) 42563779299
# Here, the unit byte 
42563779299/1024/1024/1024=40G

By looking at , This redis The maximum memory of a node is 60G, But this Key It takes up 40GB,
Big enough key 了
Sure big key, And then there's the solution ：
commonly list Treat it as a queue Producer consumer Usage of ：
So our solution was ：
1: Increase consumer speed , send key Of Value Reduce
2:key Is it still in use , Do not use to clean up

3. After communication with development , Good luck , This function just migrated to the new logic , That's it key A: no, no .
Then we just delete this key Just fine .
Be careful
Don't directly del Big key, have access to ltrim Delete slowly , Also pay attention to the load of the system .

result

You can see that the last deletion is directly from 85% Use down to 18%, clear as daylight

So far, the problem has been solved .