Keys and scan based on redis delete keys with TTL -1

Preface ： because redis The maximum memory used by the server is 450MB,redis Stored parts key Expiration time is not set , The memory elimination strategy is ：noeviction The above causes redis The service could not be loaded into the new key, Force some businesses to be blocked .

One 、keys Command Introduction

redis KEYS The basic syntax of the command is as follows ：

redis 127.0.0.1:6379> KEYS PATTERN

Return value

That fits the given pattern key list (Array).

# obtain  redis  All of the  key  It can be used  *.
redis 127.0.0.1:6379> KEYS *
1) "runoob3"
2) "runoob1"
3) "runoob2"

because Redis Is a single thread processing user commands , and Keys The command will traverse all at once Key, So in During command execution , Unable to execute other commands . This leads to if Redis Medium key More , that Keys Command execution time will be longer , To block Redis.

# Looking to  runoob  For the beginning of the  key：
redis 127.0.0.1:6379> KEYS runoob*
1) "runoob3"
2) "runoob1"
3) "runoob2"

Keys The shortcomings of ：

• No, limit, We can only get all qualified... At one time key, If there are millions of results , Then what is waiting for you is “ inexhaustible ” String output .

• keys The command is a traversal algorithm , The time complexity is O(N). As we just said , This command can easily lead to Redis Service carton . therefore , We should try to avoid using this command in the production environment .

Two 、scan Command Introduction

Redis Scan The command iterates over the database keys in the database .

SCAN Command is a cursor based iterator , After each call , Will return a new cursor to the user , The user needs to use this new cursor as the SCAN Cursor parameters for the command , To continue the previous iteration process .

SCAN Returns an array of two elements , The first element is a new cursor for the next iteration , The second element is an array , This array contains all the elements that are iterated . If the new cursor returns 0 Indicates that the iteration is over .

redis Scan The basic syntax of the command is as follows ：

SCAN cursor [MATCH pattern] [COUNT count]
 Parameters ：
cursor -  The cursor .
pattern -  Matching patterns .
count -  Specifies how many elements to return from the dataset , The default value is  10 .

 Return value ： Array list .

Scan The command is to traverse this one-dimensional array . The cursor value returned each time is also the index of the array .Count The parameter indicates how many elements of the array are traversed , Return the qualified results attached under these elements . Because the linked list attached under each element has different sizes , So the number of results returned each time is different .

#  Use  0  As cursor , Start a new iteration 
redis 127.0.0.1:6379> scan 0   
1) "17" #  The cursor returned in the first iteration 
2)  1) "key:12"
    2) "key:8"
    3) "key:4"
    4) "key:14"
    5) "key:16"
    6) "key:17"
    7) "key:15"
    8) "key:10"
    9) "key:3"
   10) "key:7"
   11) "key:1"

# The cursor returned in the first iteration is used  17  Start a new iteration 
redis 127.0.0.1:6379> scan 17  
1) "0"
2) 1) "key:5"
   2) "key:18"
   3) "key:0"
   4) "key:2"
   5) "key:19"
   6) "key:13"
   7) "key:6"
   8) "key:9"
   9) "key:11"

Scan Characteristics ：

1. The complexity is O(n), But it's done step by step with cursors , Will not block threads ;

2. Provide limit Parameters , You can control the maximum number of results returned each time ,limit Just one. hint, More or less results can be returned ;

3. Same as keys equally , It also provides pattern matching ;

4. The server does not need to save state for cursors , The only state of a cursor is scan Cursor integer returned to the client ;

5. The returned results may be repeated , Need client to repeat , This is very important ;

6. If there is data modification during traversal , It is uncertain whether the modified data can be traversed ;

7. Just because the result of a single return is empty doesn't mean the end of the traversal , It depends on whether the returned cursor value is zero

count Parameters, ：

• When count Parameter specified as 100 when , However redis in key There are millions of hours , At this time, the return time will be very long ,

• count After the parameter is increased , Reduce the number of interactions , The return time will be reduced .

• Count The greater the parameters ,Redis The longer the blocking time , Need to choose between .

• Limit a little ,Count Parameters and total Key Count one ,Scan The command is like Keys The effect is the same .

Count Size and Scan The relationship between total time consumption is shown in the figure below ：

Conclusion ：

1. You can find Count The bigger it is , The shorter the total time , However, the later the promotion, the less obvious it is .

2. So recommended Count The size is 1W about .

3. If you don't think about Redis The block , Actually Keys Than Scan It's going to be a lot faster , After all, one-time processing , Eliminates redundant interactions .

3、 ... and 、keys Command deletion

#!/usr/bin/python3
import redis

r = redis.Redis(host='172.18.158.92', port=6379, db=0,decode_responses=True)
var = 0
var1 = 0
list_keys = r.keys("system_WXMINI/WX_MINI_NO_Userinfo/unionid*")

for key in list_keys:
    num = r.ttl(key)
    if num == -1:
        r.delete(key)
        var = var + 1
    else:
        var1 = var1 + 1

print("end")
print(" Delete key The number of ",var)
print(" Undeleted quantity ",var1)

Four 、scan Command deletion

[[email protected] scripts]# cat clean_key_v5.py 
#!/usr/bin/python3
import redis

def RedisScan(vague_key,host="127.0.0.1",port=6379,password=None,db=0):
    redis_cache = redis.Redis(host=host, port=port, db=db, password=password, decode_responses=True)
    begin_pos,counts,var,delete_key = 0,0,0,0

    while True:
        begin_pos,list_keys = redis_cache.scan(begin_pos,vague_key,10000)
        counts += len(list_keys)
        for key in list_keys:
            num = redis_cache.ttl(key)
            if num == -1:
                redis_cache.delete(key)
                delete_key = delete_key + 1
            else:
                var = var + 1
        if begin_pos == 0:
            break

    print("no delete key is ", var)
    print("delete key is ", delete_key)
    print("total key is ", counts)

# call 
RedisScan("system_url_id*","172.18.158.92",6379,"*****")

5、 ... and 、redis Relevant adjustment

CONFIG SET maxmemory-policy volatile-lru

config set maxmemory 8589934592