MySQL large table joint query optimization, large transaction optimization, avoiding transaction timeout, lock wait timeout and lock table

background ： Recently, I made some optimization while doing the project , Mainly aimed at mysql The big table （2 Billion + When the data volume is not divided into databases and tables ） As well as the optimization of some transaction timeouts and lock waiting timeouts in production , Share some of my personal optimization ideas , Just talk about ideas without posting code .
One 、 Large table query optimization

Optimization idea ：

1. Business code level

1） Sort out the business code , Whether there are repeated or circular query databases or remote api call . If such code exists , Whether it is possible to avoid repeating redundant and cyclic time-consuming operations （ Reduce time-consuming and database connection times ）, If this kind of code appears in a transaction , It may cause transaction timeout and lock waiting timeout .

2.sql level

1）explain View execution plan , Focus on the following columns ：

id： The bigger the value is. , The higher the priority , Execute first .

table： Statement query table

type： Query type , Key fields , For viewing sql Whether it's indexed or not , What index did you go .

Performance from excellent to low ：system > const > eq_ref > ref > fulltext > ref_or_null > index_merge > unique_subquery > index_subquery > range > index > ALL

possible_keys ： Query possible indexes

key ： Index actually used , by null Indicates that no index is used

rows： Find the number of rows to scan , The smaller the value, the better

2） If you scan the whole table , Consider adding indexes . If there is an index miss , Consider changing sql How to write it , Such as query conditions to make it hit . For range queries , There may be a time when the amount of data is small mysql The optimizer will choose to index , When the amount of data is large, scan the whole table , This is a mysql The optimizer is working , At this time, if the query conditions are fixed , Consider using force (index) Force Index , It is not recommended to force an index when conditions change , Because the index may not be used under other query conditions .

3） Small tables drive large tables , If the join table query , Note that small tables drive large tables . I optimized the original sql Big table drives small table ,2 Yilian 2 Ten million .... After optimization, small tables drive large tables , Improved performance

4） Avoid index failure , Such as in,or,!= Index failure caused by such conditions

5） In case of joint index , Such as (a,b,c), Avoid query only b,bc and c The condition of results in the invalidation of the joint index

6） Try to avoid sql Make statistics in , Try to realize statistics in the code

Two 、 Big business optimization

1. Reduce transaction granularity

spring Medium @Transactional Annotations act on methods , If there are a lot of other time-consuming operations inside the method besides transaction operations , Such as a large number of time-consuming queries , It is extremely easy to cause transaction timeout and even lock wait timeout . scene ： There are a lot of time-consuming operations inside the method except transactions , The transaction timed out , Or the previous transaction has not been committed , When the latter transaction waits for the previous transaction to release the lock, it will report Lock wait timeout exceeded; try restarting transaction abnormal . terms of settlement ： Increase lock wait time , Or use programmatic transactions （TransactionTemplate） Reduce transaction granularity , Focus on time-consuming non transactional operations outside the transaction , To use less @Transactional, To use less @Transactional, To use less @Transactional.

2. Try to avoid remote calls in transactions

May exist in A The transaction of the service needs to call B Service api get data , However, the response time may be long due to network instability and other reasons , At this time, this affair may be forced to become a big affair , Even if it has only one update operation .

Not just remote calls , And message queuing , Cache, etc , Remember to put it outside the business .

This kind of code is not placed in transactions, and data consistency needs to be guaranteed , This requires the establishment of compensation mechanisms and logging ： For example, the retransmission compensation of message queue .

3. Avoid processing large amounts of data at one time in one transaction （ Such as batch or cyclic update 1000 Data ）

Do not cycle through transactions update or insert, Or operate a large amount of data in batches at one time . This may cause lock waiting timeout and lock table , Cause production accidents . Put unnecessary database operations outside the transaction , Such as log insertion

4. Some businesses are separated to do asynchronous

Consider whether some businesses in a transaction can be processed asynchronously , Such as log records , Asynchronous operations such as coupon sending , Consider asynchronous processing . Observer mode implementation or message queue implementation , Pay attention to the compensation mechanism , If a user fails to issue coupons, it can compensate for sending