A troubleshooting process for database timeout caused by SQL misguided

The phenomenon

A project I just took over recently , At noon one day 12 spot 30 Receive more development feedback , When their business requests a database , There are a lot of request timeouts , Although there have been before , But this one lasted a long time , I recovered in about two minutes .
Database version 8.0.22. Isolation level RR.

Preliminary analysis

Receive questions , take it easy , Take a look at the surveillance :


From the monitoring diagram, it can be seen that the corresponding 12 spot 36 The number of connections began to soar around the minute , And there is a small amount of slowness at this point sql.
View the slow query log at the corresponding time point , The following slow queries are found at the corresponding time sql：

Here you can see the slow query from 12 spot 36 It's done 155s.
This SQL It's not supposed to be slow , because (uid,msg_id) It's a uk Indexes ,id It's the primary key .
And then because of the timeout , So guess it is the timeout caused by the lock ？
However, the new project does not have a very deep lock related monitoring , You can only query the storage engine information
Show engine innodb status\G; You can only see the latest deadlock information . To see the complete deadlock information Suggest opening innodb_print_all_deadlocks, This parameter will output all deadlock information to MySQL Of error log among . In this way, the deadlock log at the corresponding time point will not be missed .
As a result, a deadlock message occurred at the corresponding time point , as follows ：

From the latest deadlock log , We can see ：
At that time, there was a lock waiting situation Here we need to declare the difference between lock waiting and deadlock
1. An error will be reported immediately when a deadlock occurs , And the rollback cost is small SQL. Lock waiting is based on innodb_lock_wait_timeout Default 50s, If in 50s Can't get the corresponding lock , An error will be reported try again. Therefore, the deadlock must be caused by the lock waiting first .

2. At that time, lock competition mainly focused on the primary key index record Lock contention leads to however , From two SQL Look at , these two items. SQL All have uk Indexed , Finally, the primary key cannot have lock contention .

Here we are. , I have no idea , Later, I thought I could only look back at the implementation plan , Whether the implementation plan has not reached ;

A magical scene happened , This SQL I went to the primary key index , Then there is a train of thought .
The data in the table is 2500w strip , It's almost like scanning the whole table ,
If it is a full table scan , Lock all primary keys , That makes sense .

summary

The process of the problem is as follows :
12 spot 36 branch , This is slow SQL Due to the wrong index , With all the primary key indexes gap and record lock
Then other requests come , because innodb_lock_wait_timeout=50s By default ,
So the following SQL Waiting when you can't get the lock 50s Later error reporting try again, The business response is timeout .
Thus, during the slow query execution , The corresponding in the back SQL Can't get the lock , All in 50s Time out after ,
Also because of waiting , So the connection stack , soar , Slow behind SQL 155s completion of enforcement , Back to normal .