Alibaba cloud redis development specification

Reprinted from :  Alibaba cloud Redis The development of specification - Alicloud developer community

Reprint please indicate the source of the original .

brief introduction ：  This article introduces the use of Alibaba cloud Redis Development specifications for , From key value design 、 Command to use 、 Client side usage 、 Relevant tools, etc , Through the introduction of this article, we can reduce the use of Redis Problems with the process .

One 、 Key design

1. key Name Design

• (1)【 Suggest 】: Readability and manageability

In the name of business ( Or database name ) The prefix ( prevent key Conflict ), Separate... With a colon , Such as business name : Table name :id

ugc:video:1

• (2)【 Suggest 】： Conciseness

Under the premise of semantic guarantee , control key The length of , When key More time , Memory usage should not be ignored , for example ：

user:{uid}:friends:messages:{mid} Simplified as u:{uid}:fr:m:{mid}.

• (3)【 mandatory 】： Don't include special characters

Counter example ： Include spaces 、 Line break 、 Single and double quotes and other escape characters

Detailed analysis

2. value Design

• (1)【 mandatory 】： Refuse bigkey( Prevent network card traffic 、 The slow query )

string The type is controlled in 10KB within ,hash、list、set、zset The number of elements should not exceed 5000.

Counter example ： A contain 200 Ten thousand elements list.

Not string bigkey, Do not use del Delete , Use hscan、sscan、zscan Delete gradually , At the same time, pay attention to prevent bigkey The expiration time automatically deletes the problem ( For example, one 200 Ten thousand zset Set up 1 Hours expired , Will trigger del operation , Cause obstruction , And the operation will not not not appear in slow query (latency having evidence or referent sources )), To find the way and Delete method

Detailed analysis

• (2)【 recommend 】： Choose the right data type .

for example ： Entity type ( To control and use data structure memory coding to optimize the configuration , for example ziplist, But also pay attention to the balance between memory saving and performance )

Counter example ：

set user:1:name tom
set user:1:age 19
set user:1:favor football

Example :

hmset user:1 name tom age 19 favor football

3.【 recommend 】： control key Life cycle of ,redis It's not a trash can .

It is recommended to use expire Set expiration time ( Conditions allow you to break the expiration date , Prevent concentration from expiring ), Non overdue data focuses on idletime.

Two 、 Command to use

1.【 recommend 】 O(N) Command attention N The number of

for example hgetall、lrange、smembers、zrange、sinter It is not impossible to use , But it needs to be clear N Value . There are traversal requirements to use hscan、sscan、zscan Instead of .

2.【 recommend 】： To disable the command

Do not use... Online keys、flushall、flushdb etc. , adopt redis Of rename The mechanism forbids orders , Or use scan In a gradual way .

3.【 recommend 】 The rational use of select

redis Many databases are weak , Use numbers to distinguish , Many clients have poor support , At the same time, multiple services are actually handled by multiple databases or single threads , There will be interference .

4.【 recommend 】 Use batch operation to improve efficiency

 Native command ： for example mget、mset.
 Non native command ： have access to pipeline Increase of efficiency .

But pay attention to the control of a batch operation Element number ( for example 500 within , It's also related to the number of bytes in the element ).

Notice the difference ：

1.  Primitive is atomic operation ,pipeline It's non atomic operation .
2. pipeline Different orders can be packed , Native can't do 
3. pipeline Need client and server support at the same time .

5.【 Suggest 】Redis Transaction function is weak , It is not recommended to use too much

Redis The transaction function of is weak ( Rollback is not supported ), And the cluster version ( Self research and official ) Requiring a transaction operation key Must be in a slot On ( have access to hashtag Functional solution )

6.【 Suggest 】Redis The cluster version is using Lua There are special requirements on ：

• 1. all key All should be done by KEYS Array to pass ,redis.call/pcall It's called redis command ,key The location of , Must be KEYS array, Otherwise go straight back to error,"-ERR bad lua script for redis cluster, all the keys that the script uses should be passed using the KEYS array"
• 2. all key, Must be in 1 individual slot On , Otherwise go straight back to error, "-ERR eval/evalsha command keys must in same slot"

7.【 Suggest 】 Use... If necessary monitor On command , Be careful not to use it for a long time .

3、 ... and 、 Client side usage

1.【 recommend 】

Avoid using one... For multiple applications Redis example

Example ： Unrelated business split , Make public data service .

2.【 recommend 】

Use a database with a connection pool , Can effectively control the connection , Improve efficiency at the same time , Standard usage ：

 Execute the order as follows ：
Jedis jedis = null;
try {
    jedis = jedisPool.getResource();
    // Specific orders 
    jedis.executeCommand()
} catch (Exception e) {
    logger.error("op key {} error: " + e.getMessage(), key, e);
} finally {
    // Note that this is not a closed connection , stay JedisPool In mode ,Jedis Will be returned to the resource pool .
    if (jedis != null) 
        jedis.close();
}

Here is JedisPool Optimization method article :

• Jedis Summary of common exceptions
• JedisPool Resource pool optimization

3.【 Suggest 】

High and send suggestions to the client to add fuse function ( for example netflix hystrix)

4.【 recommend 】

Set a reasonable password , Use... If necessary SSL Encrypted access （ Alibaba cloud Redis Support ）

5.【 Suggest 】

Depending on your business type , Choose well maxmemory-policy( Maximum memory retirement strategy ), Set expiration time .

The default policy is volatile-lru, After exceeding the maximum memory , Use... In expiration key lru Algorithm key The elimination of , Guarantee that the data will not be deleted , But there may be OOM problem .

Other strategies are as follows ：

• allkeys-lru： according to LRU Algorithm delete key , Whether or not the timeout property is set for the data , Until there is enough space .
• allkeys-random： Randomly delete all keys , Until there is enough space .
• volatile-random: Randomly delete expiration key , Until there is enough space .
• volatile-ttl： According to the ttl attribute , Delete data that is about to expire recently . without , Back to noeviction Strategy .
• noeviction： No data will be excluded , Reject all writes and return client error messages "(error) OOM command not allowed when used memory", here Redis Only respond to read operations .

Four 、 Related tools

1.【 recommend 】： Data synchronization

redis Data synchronization between can be used ：redis-port

2.【 recommend 】：big key Search for

redis Big key Search tools

3.【 recommend 】： hotspot key seek ( Internal implementation uses monitor, So it is recommended to use )

facebook Of redis-faina

 Alibaba cloud Redis Hot spots have been solved at the kernel level key problem , Welcome to use .

5、 ... and appendix ： Delete bigkey

1.  The following operations can be used pipeline Speed up .
2. redis 4.0 Has supported key Asynchronous deletion of , Welcome to use .

1. Hash Delete : hscan + hdel

public void delBigHash(String host, int port, String password, String bigHashKey) {
    Jedis jedis = new Jedis(host, port);
    if (password != null && !"".equals(password)) {
        jedis.auth(password);
    }
    ScanParams scanParams = new ScanParams().count(100);
    String cursor = "0";
    do {
        ScanResult<Entry<String, String>> scanResult = jedis.hscan(bigHashKey, cursor, scanParams);
        List<Entry<String, String>> entryList = scanResult.getResult();
        if (entryList != null && !entryList.isEmpty()) {
            for (Entry<String, String> entry : entryList) {
                jedis.hdel(bigHashKey, entry.getKey());
            }
        }
        cursor = scanResult.getStringCursor();
    } while (!"0".equals(cursor));
    
    // Delete bigkey
    jedis.del(bigHashKey);
}

2. List Delete : ltrim

public void delBigList(String host, int port, String password, String bigListKey) {
    Jedis jedis = new Jedis(host, port);
    if (password != null && !"".equals(password)) {
        jedis.auth(password);
    }
    long llen = jedis.llen(bigListKey);
    int counter = 0;
    int left = 100;
    while (counter < llen) {
        // Cut it off from the left every time 100 individual 
        jedis.ltrim(bigListKey, left, llen);
        counter += left;
    }
    // Finally delete key
    jedis.del(bigListKey);
}

3. Set Delete : sscan + srem

public void delBigSet(String host, int port, String password, String bigSetKey) {
    Jedis jedis = new Jedis(host, port);
    if (password != null && !"".equals(password)) {
        jedis.auth(password);
    }
    ScanParams scanParams = new ScanParams().count(100);
    String cursor = "0";
    do {
        ScanResult<String> scanResult = jedis.sscan(bigSetKey, cursor, scanParams);
        List<String> memberList = scanResult.getResult();
        if (memberList != null && !memberList.isEmpty()) {
            for (String member : memberList) {
                jedis.srem(bigSetKey, member);
            }
        }
        cursor = scanResult.getStringCursor();
    } while (!"0".equals(cursor));
    
    // Delete bigkey
    jedis.del(bigSetKey);
}

4. SortedSet Delete : zscan + zrem

public void delBigZset(String host, int port, String password, String bigZsetKey) {
    Jedis jedis = new Jedis(host, port);
    if (password != null && !"".equals(password)) {
        jedis.auth(password);
    }
    ScanParams scanParams = new ScanParams().count(100);
    String cursor = "0";
    do {
        ScanResult<Tuple> scanResult = jedis.zscan(bigZsetKey, cursor, scanParams);
        List<Tuple> tupleList = scanResult.getResult();
        if (tupleList != null && !tupleList.isEmpty()) {
            for (Tuple tuple : tupleList) {
                jedis.zrem(bigZsetKey, tuple.getElement());
            }
        }
        cursor = scanResult.getStringCursor();
    } while (!"0".equals(cursor));
    
    // Delete bigkey
    jedis.del(bigZsetKey);
}