当前位置:网站首页>Redisson bug analysis
Redisson bug analysis
2022-07-06 19:51:00 【InfoQ】
1. background
- jedis It's based on blocking I/O, And the method calls are synchronous , Program flow needs to wait until socket processed I/O To perform , Asynchronous is not supported ,jedis Client instance is not thread safe , You need to use... Through a connection pool jedis.
- redisson Use non blocking I/O And based on Netty Event driven communication layer of framework , Its method calls are asynchronous .redisson Of API It's thread safe , You can operate a single redisson Connect to complete various operations .
2. Problem analysis
2.1 network connections
net.ipv4.tcp_keepalive_time=30
net.ipv4.tcp_keepalive_probes
net.ipv4.tcp_keepalive_intvl
net.ipv4.tcp_keepalive_intvl=15
net.ipv4.tcp_keepalive_probes=5
# Close the connection after a client is idle for N seconds (0 to disable)
timeout 0
# TCP keepalive.
#
# If non-zero, use SO_KEEPALIVE to send TCP ACKs to clients in absence
# of communication. This is useful for two reasons:
#
# 1) Detect dead peers.
# 2) Take the connection alive from the point of view of network
# equipment in the middle.
#
# On Linux, the specified value (in seconds) is the period used to send ACKs.
# Note that to close the connection the double of the time is needed.
# On other kernels the period depends on the kernel configuration.
#
# A reasonable value for this option is 60 seconds.
tcp-keepalive 60
2.2 Problem recurrence
2022-01-24 02:55:39.685 [http-nio-8080-exec-6] ERROR [o.a.c.c.C.[.[.[.[dispatcherServlet]:175] [TID:N/A] - Servlet.service() for servlet [dispatcherServlet] in context with path [/cmccplus] threw exception
org.springframework.dao.InvalidDataAccessApiUsageException: Unexpected exception while processing command; nested exception is org.redisson.client.RedisException: Unexpected exception while processing command
at org.redisson.spring.data.connection.RedissonExceptionConverter.convert(RedissonExceptionConverter.java:52)
at org.redisson.spring.data.connection.RedissonExceptionConverter.convert(RedissonExceptionConverter.java:35)
at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
at org.redisson.spring.data.connection.RedissonConnection.transform(RedissonConnection.java:195)
at org.redisson.spring.data.connection.RedissonConnection.syncFuture(RedissonConnection.java:190)
at org.redisson.spring.data.connection.RedissonConnection.sync(RedissonConnection.java:356)
at org.redisson.spring.data.connection.RedissonConnection.write(RedissonConnection.java:722)
at org.redisson.spring.data.connection.RedissonConnection.setEx(RedissonConnection.java:530)
at org.springframework.data.redis.connection.DefaultStringRedisConnection.setEx(DefaultStringRedisConnection.java:1138)
at org.springframework.data.redis.core.DefaultValueOperations$8.potentiallyUsePsetEx(DefaultValueOperations.java:337)
at org.springframework.data.redis.core.DefaultValueOperations$8.doInRedis(DefaultValueOperations.java:330)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:223)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:190)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:97)
at org.springframework.data.redis.core.DefaultValueOperations.set(DefaultValueOperations.java:325)
at com.cmcc.hy.plus.biz.auth.JwtAuthenticationTokenFilter.doFilterInternal(JwtAuthenticationTokenFilter.java:150)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
at org.apache.catalina.core.StandardHostValve.invoke$original$j3CdN2QH(StandardHostValve.java:135)
at org.apache.catalina.core.StandardHostValve.invoke$original$j3CdN2QH$accessor$lXTjUmdl(StandardHostValve.java)
at org.apache.catalina.core.StandardHostValve$auxiliary$mFQ4Aq8A.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.redisson.client.RedisException: Unexpected exception while processing command
at org.redisson.command.CommandAsyncService.convertException(CommandAsyncService.java:328)
at org.redisson.command.CommandAsyncService.get(CommandAsyncService.java:128)
at org.redisson.spring.data.connection.RedissonConnection.syncFuture(RedissonConnection.java:188)
... 50 common frames omitted
Caused by: java.lang.NullPointerException: null
at org.redisson.client.handler.CommandEncoder.encode(CommandEncoder.java:130)
at org.redisson.client.handler.CommandEncoder.encode(CommandEncoder.java:99)
at org.redisson.client.handler.CommandEncoder.encode(CommandEncoder.java:55)
at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:107)
at org.redisson.client.handler.CommandEncoder.write(CommandEncoder.java:75)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:120)
at org.redisson.client.handler.CommandBatchEncoder.write(CommandBatchEncoder.java:45)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:801)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:814)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:794)
at org.redisson.client.handler.CommandsQueue.write(CommandsQueue.java:84)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
at io.netty.channel.ChannelDuplexHandler.write(ChannelDuplexHandler.java:106)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38)
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1081)
at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128)
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1070)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
[redisson-timer-4-1] ERROR [o.r.c.handler.PingConnectionHandler:77] [TID: N/A] - Unable to send PING command over channel: [id: 0x3564ed6a, L:/2.0.4.28:61450 - R:172.21.41.10/172.21.41.10:6379]
org.redisson.client.RedisTimeoutException: Command execution timeout for command: (PING), params: [], Redis client: [addr=redis://172.21.41.10:6379]
at org.redisson.client.RedisConnection.lambda$async$1(RedisConnection.java:251)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:670)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:745)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:473)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
private void sendPing(ChannelHandlerContext ctx) {
RedisConnection connection = RedisConnection.getFrom(ctx.channel());
RFuture<String> future;
if (connection.getUsage() == 0) {
future = connection.async(StringCodec.INSTANCE, RedisCommands.PING);
} else {
future = null;
}
config.getTimer().newTimeout(timeout -> {
if (connection.isClosed() || ctx.isRemoved()) {
return;
}
if (connection.getUsage() == 0
&& future != null
&& (future.cancel(false) || !future.isSuccess())) {
ctx.channel().close();
if (future.cause() != null && !future.isCancelled()) {
log.error("Unable to send PING command over channel: " + ctx.channel(), future.cause());
}
log.debug("channel: {} closed due to PING response timeout set in {} ms", ctx.channel(), config.getPingConnectionInterval());
} else {
sendPing(ctx);
}
}, config.getPingConnectionInterval(), TimeUnit.MILLISECONDS);
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
RedisConnection connection = RedisConnection.getFrom(ctx.channel());
if (connection != null) {
connection.fireDisconnected();
if (!connection.isClosed()) {
if (connection.isFastReconnect()) {
tryReconnect(connection, 1);
} else {
reconnect(connection, 1);
}
}
}
ctx.fireChannelInactive();
}
2.3 Source code analysis
<T> T write(byte[] key, Codec codec, RedisCommand<?> command, Object... params) {
RFuture<T> f = executorService.writeAsync(key, codec, command, params);
indexCommand(command);
return sync(f);
}
executorService.writeAsync
sync
RedisCommand
params
writeAsync
CommandEncoder
CommandEncoder
MessageToByteEncoder
@Override
protected void encode(ChannelHandlerContext ctx, CommandData<?, ?> msg, ByteBuf out) throws Exception {
try {
out.writeByte(ARGS_PREFIX);
int len = 1 + msg.getParams().length;
if (msg.getCommand().getSubName() != null) {
len++;
}
out.writeCharSequence(Long.toString(len), CharsetUtil.US_ASCII);
out.writeBytes(CRLF);
writeArgument(out, msg.getCommand().getName().getBytes(CharsetUtil.UTF_8));
if (msg.getCommand().getSubName() != null) {
writeArgument(out, msg.getCommand().getSubName().getBytes(CharsetUtil.UTF_8));
}
for (Object param : msg.getParams()) {
ByteBuf buf = encode(param);
writeArgument(out, buf);
if (!(param instanceof ByteBuf)) {
buf.release();
}
}
if (log.isTraceEnabled()) {
String info = out.toString(CharsetUtil.UTF_8);
if (RedisCommands.AUTH.equals(msg.getCommand())) {
info = info.substring(0, info.indexOf(RedisCommands.AUTH.getName()) + RedisCommands.AUTH.getName().length()) + "(password masked)";
}
log.trace("channel: {} message: {}", ctx.channel(), info);
}
} catch (Exception e) {
msg.tryFailure(e);
throw e;
}
}
encode
msg.tryFailure
try {
stringRedisTemplate.opsForValue().set("test-null", null);
} catch (Exception e) {
log.error("write null", e);
}
try {
stringRedisTemplate.opsForValue().set("test-normal-write", "1");
} catch (Exception e) {
log.error("write normal", e);
}
org.springframework.dao.QueryTimeoutException: Redis server response timeout (3000 ms) occured after 0 retry attempts. Increase nettyThreads and/or timeout settings. Try to define pingConnectionInterval setting. Command: (SET), params: [[116, 101, 115, 116, 45, 110, 111, 114, 109, 97, ...], [49]], channel: [id: 0x5fc67cb9, L:/2.0.4.190:55103 - R:172.21.41.10/172.21.41.10:6379]; nested exception is org.redisson.client.RedisResponseTimeoutException: Redis server response timeout (3000 ms) occured after 0 retry attempts. Increase nettyThreads and/or timeout settings. Try to define pingConnectionInterval setting. Command: (SET), params: [[116, 101, 115, 116, 45, 110, 111, 114, 109, 97, ...], [49]], channel: [id: 0x5fc67cb9, L:/2.0.4.190:55103 - R:172.21.41.10/172.21.41.10:6379]
at org.redisson.spring.data.connection.RedissonExceptionConverter.convert(RedissonExceptionConverter.java:48)
at org.redisson.spring.data.connection.RedissonExceptionConverter.convert(RedissonExceptionConverter.java:35)
at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
at org.redisson.spring.data.connection.RedissonConnection.transform(RedissonConnection.java:195)
at org.redisson.spring.data.connection.RedissonConnection.syncFuture(RedissonConnection.java:190)
at org.redisson.spring.data.connection.RedissonConnection.sync(RedissonConnection.java:356)
at org.redisson.spring.data.connection.RedissonConnection.write(RedissonConnection.java:722)
at org.redisson.spring.data.connection.RedissonConnection.set(RedissonConnection.java:490)
at org.springframework.data.redis.connection.DefaultStringRedisConnection.set(DefaultStringRedisConnection.java:1102)
at org.springframework.data.redis.core.DefaultValueOperations$7.inRedis(DefaultValueOperations.java:309)
at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:61)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:223)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:190)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:97)
at org.springframework.data.redis.core.DefaultValueOperations.set(DefaultValueOperations.java:305)
at com.cmcc.hy.plus.job.controller.TestController.redis(TestController.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at com.cmcc.hy.plus.common.filter.LoggingFilter.doFilterInternal(LoggingFilter.java:243)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1732)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.redisson.client.RedisResponseTimeoutException: Redis server response timeout (3000 ms) occured after 0 retry attempts. Increase nettyThreads and/or timeout settings. Try to define pingConnectionInterval setting. Command: (SET), params: [[116, 101, 115, 116, 45, 110, 111, 114, 109, 97, ...], [49]], channel: [id: 0x5fc67cb9, L:/2.0.4.190:55103 - R:172.21.41.10/172.21.41.10:6379]
at org.redisson.command.RedisExecutor.lambda$scheduleResponseTimeout$5(RedisExecutor.java:341)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:670)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:745)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:473)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
org.redisson.command.RedisExecutor#scheduleResponseTimeout
TimerTask timeoutResponseTask = timeout -> {
if (isResendAllowed(attempt, attempts)) {
if (!attemptPromise.cancel(false)) {
return;
}
connectionManager.newTimeout(t -> {
attempt++;
if (log.isDebugEnabled()) {
log.debug("attempt {} for command {} and params {}",
attempt, command, LogHelper.toString(params));
}
mainPromiseListener = null;
execute();
}, retryInterval, TimeUnit.MILLISECONDS);
return;
}
attemptPromise.tryFailure(
new RedisResponseTimeoutException("Redis server response timeout (" + timeoutAmount + " ms) occured"
+ " after " + attempt + " retry attempts. Increase nettyThreads and/or timeout settings. Try to define pingConnectionInterval setting. Command: "
+ LogHelper.toString(command, params) + ", channel: " + connection.getChannel()));
};
timeout = connectionManager.newTimeout(timeoutResponseTask, timeoutTime, TimeUnit.MILLISECONDS);
scheduleResponseTimeout
org.redisson.command.RedisExecutor#checkWriteFuture
checkWriteFuture
connectionFuture.onComplete((connection, e) -> {
if (connectionFuture.isCancelled()) {
connectionManager.getShutdownLatch().release();
return;
}
if (!connectionFuture.isSuccess()) {
connectionManager.getShutdownLatch().release();
exception = convertException(connectionFuture);
return;
}
sendCommand(attemptPromise, connection);
writeFuture.addListener(new ChannelFutureListener() {
@Override
public void operationComplete(ChannelFuture future) throws Exception {
checkWriteFuture(writeFuture, attemptPromise, connection);
}
});
});
writeFuture
checkWriteFuture
scheduleResponseTimeout
CommandEncoder
org.redisson.client.handler.RedisChannelInitializer#initChannel
@Override
protected void initChannel(Channel ch) throws Exception {
initSsl(config, ch);
if (type == Type.PLAIN) {
ch.pipeline().addLast(new RedisConnectionHandler(redisClient));
} else {
ch.pipeline().addLast(new RedisPubSubConnectionHandler(redisClient));
}
ch.pipeline().addLast(
connectionWatchdog,
CommandEncoder.INSTANCE,
CommandBatchEncoder.INSTANCE);
if (type == Type.PLAIN) {
ch.pipeline().addLast(new CommandsQueue());
} else {
ch.pipeline().addLast(new CommandsQueuePubSub());
}
if (pingConnectionHandler != null) {
ch.pipeline().addLast(pingConnectionHandler);
}
if (type == Type.PLAIN) {
ch.pipeline().addLast(new CommandDecoder(config.getAddress().getScheme()));
} else {
ch.pipeline().addLast(new CommandPubSubDecoder(config));
}
ch.pipeline().addLast(new ErrorsLoggingHandler());
config.getNettyHook().afterChannelInitialization(ch);
}
pingConnectionHandler
CommandEncoder
connectionWatchdog
ErrorsLoggingHandler -> CommandsQueue -> CommandBatchEncoder -> CommandEncoder
RedisConnectionHandler -> ConnectionWatchdog -> PingConnectionHandler -> CommandDecoder -> ErrorsLoggingHandler
CommandEncoder
CommandDecoder
CommandDecoder
protected QueueCommand getCommand(ChannelHandlerContext ctx) {
Queue<QueueCommandHolder> queue = ctx.channel().attr(CommandsQueue.COMMANDS_QUEUE).get();
QueueCommandHolder holder = queue.peek();
if (holder != null) {
return holder.getCommand();
}
return null;
}
QueueCommandHolder
CommandsQueue
@Override
public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) throws Exception {
if (msg instanceof QueueCommand) {
QueueCommand data = (QueueCommand) msg;
QueueCommandHolder holder = new QueueCommandHolder(data, promise);
Queue<QueueCommandHolder> queue = ctx.channel().attr(COMMANDS_QUEUE).get();
while (true) {
if (lock.compareAndSet(false, true)) {
try {
queue.add(holder);
ctx.writeAndFlush(data, holder.getChannelPromise());
} finally {
lock.set(false);
}
break;
}
}
} else {
super.write(ctx, msg, promise);
}
}
protected void sendNext(Channel channel) {
Queue<QueueCommandHolder> queue = channel.attr(CommandsQueue.COMMANDS_QUEUE).get();
queue.poll();
state(null);
}
CommandEncode
CommandDecoder
CommandsQueue
CommandEncoder
QueueCommand
PinConnectionHandler
ConnectionWatchDog
- and jedis equally , stay set Wait for the pre emptive check before the operation ;
- stay
CommandEncoder
Of encode When an exception is thrown , Use final Statement will queue MediumQueueCommandHolder
Introduction .
3. summary
边栏推荐
- 数据的同步为每个站点创建触发器同步表
- Leetcode brush first_ Maximum Subarray
- A5000 vgpu display mode switching
- 1805. 字符串中不同整数的数目
- 腾讯T3大牛手把手教你,大厂内部资料
- 【翻译】Linkerd在欧洲和北美的采用率超过了Istio,2021年增长118%。
- 腾讯安卓开发面试,android开发的基础知识
- 350. Intersection of two arrays II
- Interview assault 63: how to remove duplication in MySQL?
- Selenium advanced operations
猜你喜欢
[translation] micro survey of cloud native observation ability. Prometheus leads the trend, but there are still obstacles to understanding the health of the system
In depth analysis, Android interview real problem analysis is popular all over the network
某东短信登录复活 安装部署教程
A5000 vgpu display mode switching
Information System Project Manager - Chapter VIII project quality management
利用 clip-path 绘制不规则的图形
Teach you to learn JS prototype and prototype chain hand in hand, a tutorial that monkeys can understand
[infrastructure] deployment and configuration of Flink / Flink CDC (MySQL / es)
腾讯T3大牛手把手教你,大厂内部资料
OceanBase社区版之OBD方式部署方式单机安装
随机推荐
PHP and excel phpexcel
How to do smoke test
【翻译】数字内幕。KubeCon + CloudNativeCon在2022年欧洲的选择过程
转让malloc()该功能后,发生了什么事内核?附malloc()和free()实现源
Information System Project Manager - Chapter VIII project quality management
Poj3617 best cow line
MySql必知必会学习
算法面试经典100题,Android程序员最新职业规划
Method keywords deprecated, externalprocname, final, forcegenerate
手把手教你学会js的原型与原型链,猴子都能看懂的教程
Unbalance balance (dynamic programming, DP)
Interpretation of Dagan paper
POJ 3207 Ikki&#39;s Story IV – Panda&#39;s Trick (2-SAT)
Social recruitment interview experience, 2022 latest Android high-frequency selected interview questions sharing
Monthly report of speech synthesis (TTS) and speech recognition (ASR) papers in June 2022
Alibaba data source Druid visual monitoring configuration
腾讯T3大牛手把手教你,大厂内部资料
logstash高速入口
Tencent Android interview must ask, 10 years of Android development experience
121. The best time to buy and sell stocks