当前位置：网站首页>[online problem] timeout waiting for connection from pool
[online problem] timeout waiting for connection from pool
2022-06-11 17:27:00 【new hilbert()】
1. Problem finding
background ： Conduct app When pushing from the end , Some scenarios will use mobile phone manufacturers to push , In use Vivo Of sdk When pushing ,qps When it is too high, the following problems will occur .
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:286)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute$original$2s8CgZpA(InternalHttpClient.java:184)
at org.apache.http.impl.client.InternalHttpClient.doExecute$original$2s8CgZpA$accessor$3Fb5SRfG(InternalHttpClient.java)
at org.apache.http.impl.client.InternalHttpClient$auxiliary$MIrDHOOm.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:88)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at com.vivo.push.sdk.common.HttpUtil.execute(HttpUtil.java:118)
at com.vivo.push.sdk.common.HttpUtil.doPost(HttpUtil.java:111)
at com.vivo.push.sdk.server.Sender.sendSingle(Sender.java:43)
2. Troubleshoot problems
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
This is an acquisition HttpClient The connection timeout of the connection pool is abnormal , This indicates that the connection pool is insufficient at this time .
see vivo SDK Parameters set for the connection process pool ：
ViVo SDK Set the connection pool size to 10,HttpRoute The maximum number of connections supported by routing connections is 5, This is in the same HttpRoute Route concurrency greater than 5 Under the circumstances , Timeout failed to get connection , It's going to happen Timeout waiting for connection from pool
manager = new PoolingHttpClientConnectionManager(socketFactoryRegistry);
manager.setMaxTotal(Constants.HTTP_MAX_CONNECTION);
manager.setDefaultMaxPerRoute(Constants.HTTP_MAX_ROUTE);
HTTP_MAX_CONNECTION = 10;
HTTP_MAX_ROUTE = 5;
3. Principle analysis
（1） Set connection pool parameters
The key parameters ：
- validateAfterInactivity： Idle connection lifetime
- maxTotal： Connection pool size
- defaultMaxPerRoute： Every ip+port Or the maximum number of connections supported by the domain name
public class SupplierHttpClientUtil {
private static final CloseableHttpClient HTTP_CLIENT;
private static final PoolingHttpClientConnectionManager HTTP_CLIENT_CONNECTION_MANAGER;
@Value("${supplier.connection-timeout.default:1500}")
private static int connectTimeout;
@Value("${supplier.socket-timeout.default:1500}")
private static int socketTimeout;
@Value("${supplier.connection-request-timeout.default:500}")
private static int connectionRequestTimeout;
static {
// Create a connection pool manager
HTTP_CLIENT_CONNECTION_MANAGER = new PoolingHttpClientConnectionManager(1L, TimeUnit.SECONDS);
// Connection pool size
HTTP_CLIENT_CONNECTION_MANAGER.setMaxTotal(200);
// Every ip+port Or the maximum number of connections supported by the domain name
HTTP_CLIENT_CONNECTION_MANAGER.setDefaultMaxPerRoute(100);
// Idle connection lifetime
HTTP_CLIENT_CONNECTION_MANAGER.setValidateAfterInactivity(1000);
// Default request settings
RequestConfig configBuilder = RequestConfig.custom()
// Set connection timeout
.setConnectTimeout(connectTimeout)
// Set read timeout
.setSocketTimeout(socketTimeout)
// Set the timeout to get the connection instance from the connection pool
.setConnectionRequestTimeout(connectionRequestTimeout)
.build();
// establish httpclient object
HTTP_CLIENT = HttpClients.custom()
.evictExpiredConnections()
.evictIdleConnections(10L, TimeUnit.SECONDS)
.setDefaultRequestConfig(configBuilder)
.setConnectionManager(HTTP_CLIENT_CONNECTION_MANAGER)
.disableAutomaticRetries()
.build();
}
public static CloseableHttpClient getHttpClient() {
return HTTP_CLIENT;
}
}
// Set client send
public class HwClientSender {
public static final String CHARSET = "UTF-8";
public static String post(String url, String accessToken, String postData) throws IOException {
String result = null;
CloseableHttpClient httpClient = SupplierHttpClientUtil.getHttpClient();
CloseableHttpResponse response = null;
try {
HttpPost post = new HttpPost(url);
post.addHeader("Authorization", "Bearer " + accessToken);
post.addHeader("Content-Type", "application/json; charset=UTF-8");
StringEntity s = new StringEntity(postData, CHARSET);
s.setContentEncoding(CHARSET);
s.setContentType("application/json");
post.setEntity(s);
response = httpClient.execute(post);
if (response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) {
result = EntityUtils.toString(response.getEntity());
}
return result;
} catch (Exception e) {
log.error(" doPost error data:" + postData + " ,url:" + url + " ,msg:" + e.getMessage(), e);
} finally {
try {
if (response != null) {
response.close();
}
} catch (IOException e) {
}
}
return null;
}
}
（2）httpClient.execute Finally, the bottom layer calls requestConnection For the request
public ConnectionRequest requestConnection(
final HttpRoute route,
final Object state) {
Args.notNull(route, "HTTP route");
if (this.log.isDebugEnabled()) {
this.log.debug("Connection request: " + format(route, state) + formatStats(route));
}
// according to route Get connections from the connection pool
final Future<CPoolEntry> future = this.pool.lease(route, state, null);
return new ConnectionRequest() {
@Override
public boolean cancel() {
return future.cancel(true);
}
@Override
public HttpClientConnection get(
final long timeout,
final TimeUnit tunit) throws InterruptedException, ExecutionException, ConnectionPoolTimeoutException {
return leaseConnection(future, timeout, tunit);
}
};
}
pool.lease Methods, Return is a Future,future One of the key methods is
final E leasedEntry = getPoolEntryBlocking(route, state, timeout, timeUnit, this);
according to route Get connections from the connection pool
public Future<E> lease(final T route, final Object state, final FutureCallback<E> callback) {
Args.notNull(route, "Route");
Asserts.check(!this.isShutDown, "Connection pool shut down");
return new Future<E>() {
private final AtomicBoolean cancelled = new AtomicBoolean(false);
private final AtomicBoolean done = new AtomicBoolean(false);
private final AtomicReference<E> entryRef = new AtomicReference<E>(null);
@Override
public boolean cancel(final boolean mayInterruptIfRunning) {
if (done.compareAndSet(false, true)) {
cancelled.set(true);
lock.lock();
try {
condition.signalAll();
} finally {
lock.unlock();
}
if (callback != null) {
callback.cancelled();
}
return true;
}
return false;
}
@Override
public boolean isCancelled() {
return cancelled.get();
}
@Override
public boolean isDone() {
return done.get();
}
@Override
public E get() throws InterruptedException, ExecutionException {
try {
return get(0L, TimeUnit.MILLISECONDS);
} catch (final TimeoutException ex) {
throw new ExecutionException(ex);
}
}
@Override
public E get(final long timeout, final TimeUnit timeUnit) throws InterruptedException, ExecutionException, TimeoutException {
for (;;) {
synchronized (this) {
try {
final E entry = entryRef.get();
if (entry != null) {
return entry;
}
if (done.get()) {
throw new ExecutionException(operationAborted());
}
final E leasedEntry = getPoolEntryBlocking(route, state, timeout, timeUnit, this);
if (validateAfterInactivity > 0) {
if (leasedEntry.getUpdated() + validateAfterInactivity <= System.currentTimeMillis()) {
if (!validate(leasedEntry)) {
leasedEntry.close();
release(leasedEntry, false);
continue;
}
}
}
if (done.compareAndSet(false, true)) {
entryRef.set(leasedEntry);
done.set(true);
onLease(leasedEntry);
if (callback != null) {
callback.completed(leasedEntry);
}
return leasedEntry;
} else {
release(leasedEntry, true);
throw new ExecutionException(operationAborted());
}
} catch (final IOException ex) {
if (done.compareAndSet(false, true)) {
if (callback != null) {
callback.failed(ex);
}
}
throw new ExecutionException(ex);
}
}
}
}
};
}
（3）getPoolEntryBlocking according to route Get connection from connection pool
- RouteSpecificPool<T, C, E> pool： Every route There are corresponding pool Connection pool
private E getPoolEntryBlocking(
final T route, final Object state,
final long timeout, final TimeUnit timeUnit,
final Future<E> future) throws IOException, InterruptedException, ExecutionException, TimeoutException {
Date deadline = null;
if (timeout > 0) {
deadline = new Date (System.currentTimeMillis() + timeUnit.toMillis(timeout));
}
this.lock.lock();
try {
// Get corresponding route The connection pool
final RouteSpecificPool<T, C, E> pool = getPool(route);
E entry;
for (;;) {
Asserts.check(!this.isShutDown, "Connection pool shut down");
if (future.isCancelled()) {
throw new ExecutionException(operationAborted());
}
for (;;) {
entry = pool.getFree(state);
if (entry == null) {
break;
}
if (entry.isExpired(System.currentTimeMillis())) {
entry.close();
}
if (entry.isClosed()) {
this.available.remove(entry);
pool.free(entry, false);
} else {
break;
}
}
// If you have free time, you can return directly
if (entry != null) {
this.available.remove(entry);
this.leased.add(entry);
onReuse(entry);
return entry;
}
// See if you can create a new connection
final int maxPerRoute = getMax(route);
// See how many more
final int excess = Math.max(0, pool.getAllocatedCount() + 1 - maxPerRoute);
// Take out the excess , Because this can be set dynamically maxPerRoute
if (excess > 0) {
for (int i = 0; i < excess; i++) {
final E lastUsed = pool.getLastUsed();
if (lastUsed == null) {
break;
}
lastUsed.close();
this.available.remove(lastUsed);
pool.remove(lastUsed);
}
}
// No more than , Just create a new
if (pool.getAllocatedCount() < maxPerRoute) {
final int totalUsed = this.leased.size();
final int freeCapacity = Math.max(this.maxTotal - totalUsed, 0);
if (freeCapacity > 0) {
final int totalAvailable = this.available.size();
if (totalAvailable > freeCapacity - 1) {
if (!this.available.isEmpty()) {
final E lastUsed = this.available.removeLast();
lastUsed.close();
final RouteSpecificPool<T, C, E> otherpool = getPool(lastUsed.getRoute());
otherpool.remove(lastUsed);
}
}
final C conn = this.connFactory.create(route);
entry = pool.add(conn);
this.leased.add(entry);
return entry;
}
}
boolean success = false;
try {
pool.queue(future);
this.pending.add(future);
if (deadline != null) {
success = this.condition.awaitUntil(deadline);
} else {
this.condition.await();
success = true;
}
if (future.isCancelled()) {
throw new ExecutionException(operationAborted());
}
} finally {
// In case of 'success', we were woken up by the
// connection pool and should now have a connection
// waiting for us, or else we're shutting down.
// Just continue in the loop, both cases are checked.
pool.unqueue(future);
this.pending.remove(future);
}
// check for spurious wakeup vs. timeout
if (!success && (deadline != null && deadline.getTime() <= System.currentTimeMillis())) {
break;
}
}
throw new TimeoutException("Timeout waiting for connection");
} finally {
this.lock.unlock();
}
}
边栏推荐
- 【Mysql】redo log，undo log 和binlog详解（四）
- 拜登下令强制推行零信任架构
- Bentley 使用 Authing 快速实现应用系统与身份的集成
- Oracle database merge row records, wmsys WM_ Use of the concat function and group in MySQL_ Use and comparison of concat (ID).
- LeetCode-384. 打乱数组
- Bentley uses authing to quickly integrate application system and identity
- LeetCode-1005. K 次取反后最大化的数组和
- Derivation of child numbering formula for nodes numbered I in full k-ary tree
- Don't you understand the design and principle of thread pool? Break it up and crush it. I'll teach you how to design the thread pool
- Splitting method of MySQL large tables
猜你喜欢
CLP information -5 keywords to see the development trend of the financial industry in 2022
Activity | authing's first channel cooperation activity came to a successful conclusion
Song of the sea in 5g Era
Haas, which integrates relevant technologies of Alibaba cloud, Dharma Institute and pingtouge, has officially announced the book
Science popularization genius on the left, madman on the right
Katalon Studio Enterprise
Semaphore PV operation of process interaction and its code implementation
Xie Yang, CEO of authing, was selected into Forbes' 30 under 30 Asia list in 2021
How to simplify a lot of if... Elif... Else code?
RecyclerView缓存复用解析，源码解读
随机推荐
05_特征工程—降维
Authing biweekly news: authing forum launched (4.25-5.8)
ASP. Net education OA system source code education industry OA system source code with document
LeetCode-384. Scramble array
mysql 大表的拆分方式
Hands on deep learning - multiple input and output channels in the convolution layer
Authing CEO 谢扬入选福布斯 2021 年 30 Under 30 亚洲榜单
sql server中移除key lookup书签查找
A set of ThinkPHP wechat applet mall source code with background management
Analyze which should be tested in PMP and ACP with actual cases? Which is more useful?
Port planning and APJ
论文阅读 dyngraph2vec: Capturing Network Dynamics using Dynamic Graph Representation Learning
LeetCode——42. Connected to rainwater (double pointer)
Le compte de titres de l'école de commerce kainiu est - il sécurisé? Frais d'ouverture de compte
Database backup (MySQL)
Real time myth -- real-time RTOS multitask performance analysis
Activity | authing's first channel cooperation activity came to a successful conclusion
我的CのERROR们
Authing 背后的计算哲学
ShellBrowser . NET Crack