If no code is moved, the project access speed drops significantly the next day. Case analysis

One 、 The actual situation of the case

1. Basic business of the project

A large construction logistics platform and management platform for bidding .

2. Basic application technology of the project
Fore and aft end separation , The front frame vue Back end framework Springcloud、 database MySQL、 Containers k8s.

3. Description of the accident
The previous day, the project was accessed normally, and the user was still using it , As a result, the page data could not be printed the next day , Many interface requests have timed out . Exclude the code caused by releasing the project bug outside , It also excludes accidents caused by operation and maintenance errors , I don't know .

Two 、 Accident handling process

1. Tracking code
In general , When something goes wrong, you should first directly track the code and analyze the specific causes , Add breakpoints and analyze which block is slow step by step . The result is that sql Execution is time consuming .

2. Optimize sql
Find out sql No optimization , The index has been added , and sql Is not complicated .sql The execution speed is very unstable , Sometimes fast, sometimes slow .

3. Query database status

1） Query the number of database connections ：show processlist; 

Find out sleep There are many connections , As a result, our program code occupies very little database connection memory , perform sql Very slow .

2） Query the maximum number of connections ：show variables like '%max_connection%';

3） Set the maximum number of connections ：set global max_connections=1000;   

Although the number of connections is set too much , however sleep There are still too many connections , It takes up a lot of memory space in the database ,sql The execution speed is still very slow , So we can only clear sleep Connect .

4） Clear the connection 

• restart mysql The number of service connections is cleared naturally , However, it is recommended not to restart until you have to mysql service .
• set global wait_timeout=100 whatever sleep If the connection sleep time exceeds 100 second , Will be mysql Natural termination of services .

4. Analyze the cause
Although the above has been cleared mysql Of sleep Extra long connection waiting , The implementation of the project is solved sql Slow questions . What caused it , Will it happen again , It's a question worth thinking about .

Too many database connections , And something went wrong , There must be something wrong with the project database configuration .

1） View the original configuration 

Here are the meanings of database configuration

• name： Indicates the name of your connection pool, that is, the address where you want to access the connection pool

• auth： Is the connection pool management right attribute ,Container Indicates container management

• type： It's the type of object

• driverClassName： Is the name of the database driver

• url： Is the address of the database

• username： Is the user name of login database

• password： Is the password to log in to the database

• maxIdle： Maximum free number , Maximum idle time for database connection . More than free time , Database connection
  Connections will be marked as unavailable , And then it's released . Set to 0 Means unlimited .

• MaxActive, Maximum number of database connections to the connection pool . Set to 0 Means unlimited .

• maxWait , Maximum connection setup wait time . If this time is exceeded, an exception will be received . Set to -1 Express
  unlimited .

• timeBetweenEvictionRunsMillis Dynamic retrieval duration , That is, how often I check the connection .

• minEvictableIdleTimeMillis： Set how long the unused connection will be released .

• validationQuery： Used to check whether the connection is valid sql, The requirement is a query statement .
  If validationQuery by null,testOnBorrow、testOnReturn、
  testWhileIdle It doesn't work .

• testWhileIdle： Recommended configuration is true, No performance impact , And ensure safety .
  Check when applying for connection , If the free time is greater than
  timeBetweenEvictionRunsMillis,
  perform validationQuery Check whether the connection is valid .

• testOnBorrow： Execute on connection request validationQuery Check whether the connection is valid , This configuration will degrade performance .

• testOnReturn： Execute... When returning the connection validationQuery Check whether the connection is valid , This configuration will degrade performance .

• poolPreparedStatements： Whether the cache preparedStatement, That is to say PSCache.
  PSCache Great improvement in database performance supporting cursors , for instance oracle.
  stay mysql5.5 The following versions do not PSCache function , It is recommended to close .
  The author in 5.5 Used in version PSCache, Find... Through the monitoring interface PSCache There are cache hit rate records ,
  This should be support PSCache.

• maxPoolPreparedStatementPerConnectionSize： Specify on each connection PSCache Size .

• filters： Property type is string , Configure the extension by alias ,
  Common plug-ins are ：
  For monitoring statistics filter:stat
  It's for the log filter:log4j
  defense sql Injected filter:wall

• connectionProperties： adopt connectProperties Property to open mergeSql function ; slow SQL Record

2） Analysis leads to mysql Of sleep Too many connections

What we use here is alibaba Database connection package , Our database configuration is springBoot Original configuration , In other words, it doesn't take effect .timeBetweenEvictionRunsMillis、minEvictableIdleTimeMillis The configuration didn't work ,60s The detection time exceeds 300s Invalid connection for , And release .

3） The correct configuration of Alibaba database connection package

3、 ... and 、 Summary and treatment results

mysql The execution speed is reduced because we have too many invalid connections , Occupied mysql resources . The reason is that our database configuration is not effective , Incorrect configuration results in too many invalid database connections . The solution is to clear the invalid connections in the database , restart mysql Service or set the maximum connection duration set global wait_timeout=500. Then configure the database connection of the project correctly , Mainly timeBetweenEvictionRunsMillis、minEvictableIdleTimeMillis .