当前位置：网站首页>Want to do something in production? Then try these redis commands

Want to do something in production? Then try these redis commands

2022-07-04 03:14:00 【Java geek Technology】

Want to do something in production ？ Then try these Redis command _ iteration Ah , Recently, ah fan once again has committed a crime .

Here's the thing , Some time ago, a powder company occasionally reported errors in production transactions , After a lot of investigation, the final reason is that Redis Command execution timeout .

But what is puzzling is , Production transactions only use Redis set This simple command , This command is reasonable. It can't be executed so slowly .

What is the cause of this problem ？

To find out the problem , We looked at and analyzed Redis Recent slow logs , Most of them are time-consuming keys XX*

See the prefix of the key operated by this command , A fan just found that this is his own responsible application . But a fan checks , Although their own code did not take the initiative to use keys command , But the underlying framework is used indirectly , So we have the question of today .

Question why

The application that a fan is responsible for is a management background application , Permission management uses Shiro frame , Because there are multiple nodes , You need to use distributed Session, So here we use Redis Storage Session Information .

*
Voice over ： I don't know about distributed Session , You can see what ah fan wrote before Say in one breath 4 Distributed consistency Session Realization way , It's a tough interview ~
*

because Shiro It's not directly provided Redis Storage Session Components , A fan has to use Github An open source component shiro-redis.

because Shiro The framework needs to be validated periodically Session Whether it works , therefore Shiro The underlying layer will call SessionDAO#getActiveSessions Get all Session Information .

and shiro-redis Just inherit SessionDAO This interface , For bottom use keys Command find Redis All stored Session key.

        public Set
        <
        byte[]
        > keys(byte[] pattern){
        
    checkAndInit();
        
    Set
        <
        byte[]
        > keys = null;
        
    Jedis jedis = jedisPool.getResource();
        
    try{
        
        keys = jedis.keys(pattern);
        
    }finally{
        
        jedis.close();
        
    }
        
    return keys;
        
}
       
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.

Find the cause of the problem , The solution is simpler ,github Find solutions on , Upgrade shiro-redis To the latest version .

In this version ,shiro-redis use scan Command instead of keys, To fix the problem .

        public Set
        <
        byte[]
        > keys(byte[] pattern) {
        
    Set
        <
        byte[]
        > keys = null;
        
    Jedis jedis = jedisPool.getResource();
        

        
    try{
        
        keys = new HashSet
        <
        byte[]
        >();
        
        ScanParams params = new ScanParams();
        
        params.count(count);
        
        params.match(pattern);
        
        byte[] cursor = ScanParams.SCAN_POINTER_START_BINARY;
        
        ScanResult
        <
        byte[]
        > scanResult;
        
        do{
        
            scanResult = jedis.scan(cursor,params);
        
            keys.addAll(scanResult.getResult());
        
            cursor = scanResult.getCursorAsBytes();
        
        }while(scanResult.getStringCursor().compareTo(ScanParams.SCAN_POINTER_START) > 0);
        
    }finally{
        
        jedis.close();
        
    }
        
    return keys;
        

        
}
       
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.

Although the problem has been solved successfully , But ah fan was still a little confused .

Why? keys Instructions can slow down the execution of other commands ？

Why? Keys Instruction queries can be so slow ？

Why? Scan There's no problem with instructions ？

Redis The principle of executing commands

First let's look at the first question , Why? keys Instructions can slow down the execution of other commands ？

Answer the question , Let's take a look at Redis The client executes a command ：

Want to do something in production ？ Then try these Redis command _redis_02

From the perspective of the client , There are three steps to executing a command ：

dispatch orders
Carry out orders
Return results

But this is just what the client thinks it is , But actually at the same time , There may be many clients sending commands to Redis, and Redis We all know it's a single threaded model .

In order to handle all client requests at the same time ,Redis The internal use of queues , Queued execution .

Want to do something in production ？ Then try these Redis command _ iteration _03

So it takes four steps for the client to execute a command ：

dispatch orders
Order to line up
Carry out orders
Return results

because Redis Single thread command execution , You can only take the task out of the queue in order to start execution .

as long as 3 This process is too slow to execute commands , Other tasks in the queue have to wait , This looks to external clients ,Redis It's like it's blocked , There has been no response .

So use Redis The process must not execute instructions that need to run for a long time , This may lead to Redis Blocking , Affect the execution of other instructions .

KEYS principle

Now I'm going to answer the second question , Why? Keys Instruction queries can be so slow ？

Before I answer that question , Please think about it Redis The underlying storage structure .

It doesn't matter if you don't know your friend's , You can go back to a fan's previous article 「 Interview officer Ali ：HashMap Be familiar with it. ？ well , Let's talk about it Redis Dictionary ！」.

Here a fan copies the content of the previous article ,Redis The bottom layer uses the dictionary structure , This structure and Java HashMap The bottom layer is similar to .

Want to do something in production ？ Then try these Redis command _ client _04

keys The command needs to return all that conform to the given pattern pattern Of Redis In the key , To achieve this goal ,Redis I have to go through the dictionary ht[0] Hash table underlying array , This time complexity is 「O(N)」（N by Redis in key All the numbers ）.

If Redis in key The quantity is very small , Then the execution speed will be very fast . wait until Redis key The number of them is gradually increasing , Up to a million 、 Ten million 、 Even hundreds of millions of levels , Then the execution speed will be very slow .

The following is an experiment done by ah fan , Use lua Script to Redis add 10W individual key, And then use keys Query all keys , This query will be blocked for about ten seconds .

       
        eval "for i=1,100000  do redis.call('set',i,i+1) end" 0
       
       
        1.

*
Here a fan uses Docker Deploy Redis, The performance may be slightly worse .
*

SCAN principle

Finally, let's look at the third question , Why? scan There's no problem with instructions ？

This is because scan Order a black Technology -「 Cursor based iterators 」.

Every time you call scan command ,Redis A new cursor and a certain number of key. Next time you want to continue to get the rest of key, You need to pass this cursor into scan command , To continue the previous iteration process .

simply ,scan The command uses pagination to query redis .

Here's a scan Example of an iterative process for a command ：

scan Commands use cursors , Split a full query into many times , Reduce query complexity .

although scan Command time complexity and keys equally , All are 「O(N)」, But because of scan The command only needs to return a small number of key, So the execution speed will be very fast .

Last , although scan Order to solve keys Insufficient , But it also introduces other defects ：

The same element may be returned more than once , This requires our application to increase the ability to handle duplicate elements .
If an element is added to redis, Or deleted in the iteration process , That element will be returned , Maybe not .

The above defects , We need to consider this situation in our development .

except scan outside ,redis There are several other commands for incremental iteration ：

sscan: Used to iterate the database key in the current database , For resolution smembers There may be blocking problems
hscan The command is used to iterate over key value pairs in hash keys , For resolution hgetall There may be blocking problems .
zscan: The command is used to iterate over elements in an ordered set （ Include element members and element scores ）, Used to produce zrange There may be blocking problems .

summary

Redis Use a single thread to execute operation commands , All clients send commands ,Redis Will now be put in the queue , And then take them out of the queue and execute the corresponding commands .

If any task is too slow , It will affect other tasks in the queue , In the external client's view , It's too late to get Redis Response , It looks like it's blocked .

So don't execute in production keys、smembers、hgetall、zrange This kind of instruction may cause blocking , If you really need to execute , You can use the corresponding scan Command progressive traversal , Can effectively prevent blocking problems .

< END >

If you like our article , Welcome to forward , Click "watch" to let more people see . Want to do something in production ？ Then try these Redis command _redis_05