Record a failure caused by a custom redis distributed lock

background

The enterprise and micro alarm group continuously sends out production environment error warnings , The core information of error reporting is as follows ：

redis setNX error java.lang.NumberFormatException: For input string: "null"
  at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
  at java.lang.Long.parseLong(Long.java:589)
  at java.lang.Long.parseLong(Long.java:631)
......

Locate by abnormal information , Discovery is customized in the project Redis Distributed lock error , And the exception occurred suddenly after the recent requirement was launched , And accompanied by the exception , There is also the problem of partial disorder of the business data involved in the requirements .

Problem analysis

Old rules , First post the code involved

// section 
public class RedisLockAspect{
  public void around(ProceedingJoinPoint pjp) {
    String key = "...";
    try {
      // Blocking , Until the lock is acquired 
      while (!JedisUtil.lock(key, timeOut)) {
        Thread.sleep(10);
      }
      // Execute business logic 
      pjp.proceed();
    }finally {
      JedisUtil.unLock(key);
    }
  }
}

The above is customized Redis Facets of distributed locks , Don't look at the details , Just look at the overall logic , No big problem , Let's look at the actual locking method .

public class JedisUtil{
  public static boolean lock(String key, long timeOut){
        long currentTimeMillis = System.currentTimeMillis();
        long newExpireTime = currentTimeMillis + timeOut;
        RedisConnection connection = null;
        try {
            connection = getRedisTemplate().getConnectionFactory().getConnection();
            Boolean setNxResult = connection.setNX(key.getBytes(StandardCharsets.UTF_8), String.valueOf(newExpireTime).getBytes(StandardCharsets.UTF_8));
          // Location 1
            if(setNxResult){
                expire(key,timeOut, TimeUnit.MILLISECONDS);
                return true;
            }
          // Location 2
            Object objVal = getRedisTemplate().opsForValue().get(key);
            String currentValue  = String.valueOf(objVal);
          // Location 3, The abnormal position is if In judgment Long.parseLong(currentValue),currentValue by null String 
            if (currentValue != null && Long.parseLong(currentValue) < currentTimeMillis)  {
                String oldExpireTime = (String) getAndSet(key, String.valueOf(newExpireTime));
                if (oldExpireTime != null && oldExpireTime.equals(currentValue)) {
                    return true;
                }
            }
        }
        return false;
    }
  
  public static void unLock(String key){
    getRedisTemplate().delete(key);
  }
}

Experienced boss sees this code , I guess I can't help being rude , But let's leave it alone , Look at the wrong position first .

Abnormal information can be seen ,currentValue The value of is string “null”, namely String.valueOf(objVal) Medium objVal The object is null, That is to say Redis in ,key Corresponding value non-existent , Now think about it ,key Corresponding value non-existent , There are only two cases ：

1. key Be actively deleted ;
2. key Out of date .

Continue to follow the code up , It is found that setNx command , And back to setNxResult Indicates whether it was successful . Normally , When setNxResult by false When , Locking failed , At this point, the code should not go down , But in this code , But continue to go down ！ Asked relevant colleagues , It is said that it is to make a reentrant lock ......（ Weak roast , But the re-entry lock doesn't work like this ...）

In fact, this analysis , You can already know what caused the abnormal fault , That's what it says ,key Be actively deleted 、key Due to expiration , Let's assume that there are two threads , To the same key Lock , Corresponding to the above two situations respectively ：

key Being deleted voluntarily , Occurs after the distributed locking logic is executed , call unlock Method , See above RedisLockAspect Class finally part , Here's the picture ：

key Past due , Mainly after the thread is locked and the expiration time is set , The time spent executing business code exceeds the set lock expiration time , And before the lock expires , Lock not renewed ：

Solution

From the code above , It's not simple anymore Long.parseLong("null") Problem. , This is the whole thing Redis The problem of distributed lock implementation , And the distributed lock is widely used in the whole project , It is conceivable that the problem is very serious , If it's just a solution Long.parseLong("null") The problem of , There is no doubt that it is tickling between boots , It doesn't make any sense .

In general , Customize Redis Distributed locks are prone to the following problems ：

• setNx Lock release problem ;
• setNx Expire Atomic question ;
• Lock expiration problem ;
• Multi thread lock release problem ;
• Reentrant problem ;
• Spinlock problem in case of a large number of failures ;
• Lock data synchronization under master-slave architecture ;

Combined with the above fault codes , You can find Redis The implementation of distributed locks is hardly correct Redis Consider the distributed lock problem , The following are the main problems and corresponding solutions ：

• setNx and expire Atomic manipulation ： Use Lua Script , In a Lua In the script command , perform setNx And expire command , Guaranteed atomicity ;
• Lock expiration problem ： To prevent the lock from automatically expiring , Before the lock expires , Periodically renew the lock expiration time .
• Reentrant problem ： The granularity of reentrant design needs to reach the thread level , Thread uniqueness can be added to the lock id.
• Lock spin problem ： Reference resources JDK in AQS Design , To achieve the maximum waiting time when acquiring a lock .

For the problems in the project and the solution implementation of each problem ,baidu There are a lot of references at once , No more about . At present, the more mature comprehensive solution is to use Redisson client , The following is simple pseudocode demo：

public class RedisLockAspect{
  @Autowired
  private Redisson redisson;
  
  public void around(ProceedingJoinPoint pjp) {
    String key = "...";
    Long waitTime = 3000L;
    // Get the lock 
    RLock lock = redisson.getLock(key);
    boolean lockSuccess = false;
    try {
      // Lock and set timeout , Prevent infinite spin . The watchdog function is enabled by default （ Automatically renew locks ）
      lockSuccess = lock.tryLock(waitTime);
      // Execute business logic 
      pjp.proceed();
    }finally {
      // Unlock , Prevent other thread locks from being released 
      if (lock.isLocked() && lock.isHeldByCurrentThread() && lockSuccess){
          lock.unlock();
      }
    }
  }
}

Use Redisson It can quickly solve the problems in the current project Redis Problems with distributed locks . besides , about Redis Lock problem caused by data synchronization in master-slave architecture , Corresponding solutions RedLock, The corresponding implementation is also provided .

See the official documents for more information github.com/liulongbiao…

summary

For distributed locks , The realizable scheme is far more than Redis This implementation approach , For example, based on Zookeeper、 be based on Etcd And so on , But for the purpose , They all go the same way , The point is , How to safely 、 Use these solutions correctly , Make sure the business is normal .

For the R & D team , For similar problems , Technical partners need to be trained , Keep improving technology , We need to pay more attention to codereview Work , Identify risks in a timely manner , Avoid serious loss caused by failure （ This failure caused dirty data repair to take more than a week ）.

Fear technology , Loyal to business .