当前位置:网站首页>Record a failure caused by a custom redis distributed lock
Record a failure caused by a custom redis distributed lock
2022-07-26 01:20:00 【ImportNew】
( to ImportNew Plus star , Improve Java Skill )
background
redis setNX error java.lang.NumberFormatException: For input string: "null"at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)at java.lang.Long.parseLong(Long.java:589)at java.lang.Long.parseLong(Long.java:631)......
Locate by abnormal information , Discovery is customized in the project Redis Distributed lock error , And the exception occurred suddenly after the recent requirement was launched , And accompanied by the exception , There is also the problem of partial disorder of the business data involved in the requirements .
Problem analysis
Old rules , First post the code involved :
// sectionpublic class RedisLockAspect{public void around(ProceedingJoinPoint pjp) {String key = "...";try {// Blocking , Until the lock is acquiredwhile (!JedisUtil.lock(key, timeOut)) {Thread.sleep(10);}// Execute business logicpjp.proceed();}finally {JedisUtil.unLock(key);}}}
The above is customized Redis Facets of distributed locks , Don't look at the details , Just look at the overall logic , No big problem .
public class JedisUtil{public static boolean lock(String key, long timeOut){long currentTimeMillis = System.currentTimeMillis();long newExpireTime = currentTimeMillis + timeOut;RedisConnection connection = null;try {connection = getRedisTemplate().getConnectionFactory().getConnection();Boolean setNxResult = connection.setNX(key.getBytes(StandardCharsets.UTF_8), String.valueOf(newExpireTime).getBytes(StandardCharsets.UTF_8));// Location 1if(setNxResult){expire(key,timeOut, TimeUnit.MILLISECONDS);return true;}// Location 2Object objVal = getRedisTemplate().opsForValue().get(key);String currentValue = String.valueOf(objVal);// Location 3, The abnormal position is if In judgment Long.parseLong(currentValue),currentValue by null Stringif (currentValue != null && Long.parseLong(currentValue) < currentTimeMillis) {String oldExpireTime = (String) getAndSet(key, String.valueOf(newExpireTime));if (oldExpireTime != null && oldExpireTime.equals(currentValue)) {return true;}}}return false;}public static void unLock(String key){getRedisTemplate().delete(key);}}
Experienced boss sees this code , I guess I can't help being rude , But let's leave it alone , Look at the wrong position first .
Abnormal information can be seen ,currentValue The value of is string “null”, namely String.valueOf(objVal) Medium objVal The object is null, That is to say Redis in ,key Corresponding value non-existent .
Now think about it ,key Corresponding value non-existent , There are only two cases :
key Be actively deleted
key Out of date
Continue to follow the code up , It is found that setNx command , And back to setNxResult Indicates whether it was successful .
Normally , When setNxResult by false When , Locking failed , At this point, the code should not go down , But in this code , But continue to go down !
Asked relevant colleagues , It is said that it is to make a reentrant lock ......( Weak roast , But the re-entry lock doesn't work like this ...)
In fact, this analysis , You can already know what caused the abnormal fault , That's what it says ,key Be actively deleted 、key Due to expiration .
Let's assume that there are two threads , To the same key Lock , Corresponding to the above two situations respectively :
①key Being deleted voluntarily , Occurs after the distributed locking logic is executed , call unlock Method , See above RedisLockAspect Class finally part , Here's the picture :


Solution
From the code above , It's not simple anymore Long.parseLong("null") Problem. , This is the whole thing Redis The problem of distributed lock implementation .
And the distributed lock is widely used in the whole project , It is conceivable that the problem is very serious , If it's just a solution Long.parseLong("null") The problem of , There is no doubt that it is tickling between boots , It doesn't make any sense .
In general , Customize Redis Distributed locks are prone to the following problems :
setNx Lock release problem
setNx Expire Atomic question
Lock expiration problem
Multi thread lock release problem
Reentrant problem
Spinlock problem in case of a large number of failures
Lock data synchronization under master-slave architecture
Combined with the above fault codes , You can find Redis The implementation of distributed locks is hardly correct Redis Consider the distributed lock problem .
The following are the main problems and corresponding solutions :
setNx and expire Atomic manipulation : Use Lua Script , In a Lua In the script command , perform setNx And expire command , Guaranteed atomicity .
Lock expiration problem : To prevent the lock from automatically expiring , Before the lock expires , Periodically renew the lock expiration time .
Reentrant problem : The granularity of reentrant design needs to reach the thread level , Thread uniqueness can be added to the lock id.
Lock spin problem : Reference resources JDK in AQS Design , To achieve the maximum waiting time when acquiring a lock .
For the problems in the project and the solution implementation of each problem ,baidu There are a lot of references at once , No more about .
public class RedisLockAspect{@Autowiredprivate Redisson redisson;public void around(ProceedingJoinPoint pjp) {String key = "...";Long waitTime = 3000L;// Get the lockRLock lock = redisson.getLock(key);boolean lockSuccess = false;try {// Lock and set timeout , Prevent infinite spin . The watchdog function is enabled by default ( Automatically renew locks )lockSuccess = lock.tryLock(waitTime);// Execute business logicpjp.proceed();}finally {// Unlock , Prevent other thread locks from being releasedif (lock.isLocked() && lock.isHeldByCurrentThread() && lockSuccess){lock.unlock();}}}}
Use Redisson It can quickly solve the problems in the current project Redis Problems with distributed locks . besides , about Redis Lock problem caused by data synchronization in master-slave architecture , Corresponding solutions RedLock, The corresponding implementation is also provided .
summary
For distributed locks , The realizable scheme is far more than Redis This implementation approach , For example, based on Zookeeper、 be based on Etcd And so on .
But for the purpose , They all go the same way , The point is , How to safely 、 Use these solutions correctly , Make sure the business is normal .
For the R & D team , For similar problems , Technical partners need to be trained , Keep improving technology , We need to pay more attention to codereview Work , Identify risks in a timely manner , Avoid serious loss caused by failure ( This failure caused dirty data repair to take more than a week ).
Fear technology , Loyal to business .
from : Ten years of training experience
link :https://juejin.cn/post/7113852140455460900
- EOF -
Dig three feet Redis And MySQL Data consistency issues
It's solved Redis Big key problem , His colleagues praised him for bragging
Comfortable. , Step on an unusual about distributed locks BUG!
After reading this article, there are gains ? Please forward to share with more people
Focus on 「ImportNew」, promote Java Skill

Praise and watching is the greatest support ️
边栏推荐
- Causes of signal degradation in optical fiber communication
- Linear relationship between vectors
- Half of the people in the country run in Changsha. Where do half of the people in Changsha run?
- matlab 按位 与 或 非
- ZK-Rollups工作原理
- 谷歌浏览器调试工具使用基础版(一)
- Network performance evaluation tool ping/mtr
- 代理IP服务器如何保证自身在网络中的信息安全呢
- 手游多开用模拟器多开游戏如何切换IP搬砖
- Small sample learning data set
猜你喜欢

全国一半人跑长沙,长沙一半人跑哪?

Web middleware log analysis script 3.0 (shell script)

The application and principle of changing IP software are very wide. Four methods of dynamic IP replacement are used to protect network privacy

What is informatization? What is digitalization? What are the connections and differences between the two?

Cross-lingual Transfer of Correlations between Parts of Speech and Gaze Features 阅读笔记

聚势|海泰方圆亮相第五届数字中国建设峰会

Prime Ring Problem
![[RTOS training camp] learn C language from a higher perspective](/img/4c/bbbec489abb781a1de1e99bbf12c6f.png)
[RTOS training camp] learn C language from a higher perspective

【Code】剑指offer 03数组中重复的数字

当博客被黑客攻击时该怎么办?
随机推荐
数据库系统原理与应用教程(053)—— MySQL 查询(十五):字符型函数的用法
NodeJS 基于 Dapr 构建云原生微服务应用,从 0 到 1 快速上手指南
Android SQLite first groups and then sorts left concatenated queries
Docker高级篇-Mysql主从复制
Dijkstra 求最短路
How to obtain the cash flow data of advertising services to help analyze the advertising effect?
Detailed explanation of at and crontab commands of RHCE and deployment of Chrony
[go] III. The simplest restful API server
REST-assured接口测试框架详解
Case when of SQL
[software development specification III] [software version naming Specification]
Pycharm automatically adds header comments when creating py files
机器学习:贝叶斯网络
如何获取广告服务流量变现数据,助力广告效果分析?
Half of the people in the country run in Changsha. Where do half of the people in Changsha run?
【数据挖掘】生成模型和判别模型的区别及优缺点
Cross-lingual Transfer of Correlations between Parts of Speech and Gaze Features 阅读笔记
[RTOS training camp] learn C language from a higher perspective
matlab 移位操作基础
点屏注意事项