70 root cause analysis Oracle database sudden performance problems, who will take the blame

Sudden performance problems of database , Sometimes it is possible to restart the application 、 Re collect statistics 、 Methods such as restarting the database have been temporarily solved , however , How to find the root cause of the fault , Avoid reoccurrence of the fault , It is a key step for the problem to get a perfect closed loop （ Of course , Being able to quickly restore business is also a very critical link ）. This is why many industries have high requirements for business stability （ Like finance 、 signal communication 、 The railway 、 Aviation, etc ） Database system of , Buy oracle After sales service （ACS） Top services in SSC One of the main reasons for service .

Another important aspect of root cause analysis is to find out who is responsible for the problem ： Operation and maintenance 、 Development is also the reason for the database product itself （ Defect or bug）, Some problems are unilateral , Some may have several aspects . The conclusion of root cause analysis will affect the department assessment , A conclusion that everyone is convinced , It can convince the party who carries the pot . Sometimes, let the product back to the pot , It is also a good ending .

I've seen someone share some trouble shooting articles on the Internet , Although the problem has been solved , But the conclusion of root cause analysis is completely wrong , For example, there is a conclusion that index unique scan Efficiency is far greater than index range scan The case of , Consider a database histogram limitation problem ,“ justified ” The explanation of has become the problem of index scanning method , This is a low-level error , Mislead the audience .

The case introduced today was sent by a student , Please let me know about a banking system SQL Perform root cause analysis on performance problems caused by sudden change of execution plan . I spent about half an hour analyzing this problem , The root cause is given .

Problem description ：

2019-11-21 Around 0:00 , An important business SQL The implementation plan has changed , The execution efficiency is seriously reduced . Here are the collected sqlhc Information （ Database version 11.2.0.4）：

According to the information above, we can see , Implementation plan cost It's getting smaller , But the efficiency has decreased thousands of times .

For experienced optimization experts , The first reaction to this problem is that the predicate is out of bounds , So this is the way to think about it , Get the evidence soon （ The customer thinks the information is sensitive , I won't post too many details here ）, And give the root cause .

Root cause analysis ：

1、 Through the statistical information collection history of the table, you can find ,11 month 20 Friday night , The database automatically collects statistical information on tables . Knowledge point 1： On weekdays 22：00~02：00 This time period , The system will re collect statistical information for the tables , Collect statistics .

2、 Knowledge point 2： A period of time after the collection of statistical information , It will be used for the related SQL Do hard analysis .

3、 Knowledge point 3： Hard parsing , Bound variable snooping occurs . If you peek into a bound variable , The maximum and minimum statistics on the field have been exceeded （ Usually the largest ） Range of values , Predicate out of bounds may occur . If the predicate is out of bounds, an index selection error may occur , Resulting in an inefficient execution plan .

4、 Knowledge point 4： The real predicate out of bounds usually occurs in varchar2 and number Field , Although the date field often goes out of bounds , But the system has an out of Range Algorithm for date type fields , Have a certain tolerance , It will not be considered as a real cross-border . This sql The data saved in the out of bounds field is date data , But it uses number type . This involves an important development specification ： It is highly discouraged to use number or varchar2 Type to save date data , The standard practice is to use date type . The random use of the above data types is a lazy behavior of developers in order to type fewer letters . Because when querying by day ,date Type generally needs to write two range conditions , and number or varchar2 type , An equivalent condition can be used to complete , And you don't have to do to_date transformation . This lazy and irregular practice , Will give SQL Performance brings great performance risks .

5、 Knowledge point 5： After predicate out of bounds , The database optimizer is evaluating this sql Two indexes that can be used （ Are composite indexes that begin with out of bounds fields ） when , It's not the usual way to choose , But to choose leaf blocks The less one , This case ,leaf blocks The less index , It happens to be the inefficient one .

The following figure shows the efficient index used by the normal execution plan ,sql All of the indexes are used 3 A field ; The following is the inefficient index used by the execution plan in case of performance failure ,sql It's only used. JYRQ（number Type to save date data ） A field ：

The figure below #1 Corresponding to the above efficient index ,#2 Corresponding to the inefficient index above ：

6、 Knowledge point 6：sqlhc Caught sql The implementation of the following , Many binding variables used are no longer out of bounds , And the adaptive cursor of the system （ACS） Keep it on , Why? ACS Failed to adjust the implementation plan back to normal in time ？ This involves another non-standard situation ： The data type used by bound variables is char, Just now we mentioned that the types used by fields are number, The optimizer needs to do... On the bound variables to_number Implicit type conversion , This transformation leads to ACS Don't take effect .

7、 Knowledge point 7： Many articles on the Internet say that it is necessary to close ACS, I don't quite agree with this statement , This case , although ACS Because the binding variable type does not match, it does not take effect , But if the data type used by the bound variable is also number, that ACS It will take effect , There will be no mistake in implementing the plan ： For the case that the following predicate does not cross the boundary ,ACS It can also be adjusted back to the normal implementation plan in time .ACS stay 11g Version to introduce , At the beginning of the bug More , here we are 11204 edition , quite a lot bug It has been fixed , It plays a far greater role than bug Some small problems caused by .

summary ：

Through the above analysis , Obviously, the root cause of this problem is that developers use inappropriate data types to save date data , It is very easy to cause predicate out of bounds ; At the same time, the wrong binding variable data type is used , Let the optimizer's ACS Function failure . The pot , It is entirely up to the developer to recite . however , If the operation and maintenance personnel do not master the above knowledge points , The root cause cannot be analyzed , The pot with faulty performance can only be carried by itself .

This sql, If the development does not make code adjustments , The same fault may still occur again . The temporary solution is to use sql profile Bind execution plan .

（ End ）