Troubleshooting and handling of an online problem caused by redis zadd

background

Recently, there are user feedback , The anchor received 1881 Numerical gift , Digital display under the avatar 881, Normally , Should be displayed 1881 The numerical .

After investigation , Because redis zadd The problem of data inconsistency in the case of concurrency .

Troubleshoot problems

After checking the gift giving log , Find out mongodb Data update is normal , however redis Data exception , After checking the business code, we found the problem .

The business code is as follows ：

The logic here is to update mongodb, Then on redis The numerical value is carried out zadd Cover . In general, there is no problem , But if you encounter a little high concurrency :

1. A Threads and B Threads are running simultaneously on mongo Modify , At the database level , B On A Modification of .
2. But in the process , Threads B Ahead of thread A call redis modify , Threads A Yes redis 's changes override B The numerical .
3. Lead to redis Data maintenance error in .

Problem fix

Use Lua Script modification redis value , In case of concurrency, override logic is not processed , Avoid old values overwriting new values , utilize mongodb Self accretion guarantees atomicity . The code is as follows :

Lua The script is as follows ：

SAFE_ZADD_SCRIPT = """
    local key = KEYS[1]
    local field = ARGV[1]
    local new_value = tonumber(ARGV[2])
    local old_value = redis.call("zscore", key, field)
    local res = nil
    
    if (not old_value) or (tonumber(old_value) < new_value)
    then
        res = redis.call('zadd', key, new_value, field)
    end
    
    return res
"""
 Copy code 

Why not use zincrby？

zincrby Although it is guaranteed that there will be no concurrency problems , But if key Fail or be cleaned up , It will cause the ranking list to change from 0 Start counting , Causing data errors .