Huawei cloud experts explain the new features of gaussdb (for MySQL)

This article is shared from Huawei cloud community 《

GaussDB Interpretation of innovation characteristics ：Partial Result Cache, The operator is accelerated by caching intermediate results

》, author ：GaussDB  database  .

To speed up query performance , Traditional relational database , such as Oracle、DB2, Both have the feature of result set caching , Used to cache the result set of a query statement . If the following same statement is queried , The database will get the results directly from the result set cache , Instead of re executing the query .MySQL  stay 4.0 Result set caching has also been introduced in version Query cache, But there are limitations in design , As follows ：

• Query cache For a single query , Any one of the tables has been modified , If the result set is affected, it needs to be refreshed or invalidated .

• Query cache It depends on the isolation level , Different isolation levels produce different result sets .

• Query cache All data needs to be cached , If the result of the table is large , The cache needs to occupy a large amount of memory or write to disk .

This also led to the feature being used in MySQL 8.0 Version removed .

Given the query performance gain of the result set cache , We are GaussDB（for MySQL） introduce

Partial result cache

This new feature , abbreviation PTRC. seeing the name of a thing one thinks of its function , This is also a result set caching feature . Different from traditional result set caching ,PTRC It is used to assist the execution of internal operators of a single query . in other words PTRC Smaller particle size , It caches the intermediate results of an operator within a query , So as to accelerate the operator .

there Partial  There are two layers of concepts ：

From these two points, we can see ,PTRC Is related to a single query , The lifecycle starts at the beginning of the query and ends at the end of the query , Automatic termination . Because it accelerates the operator , So a query can have multiple PTRC. As long as the optimizer calculates based on the cost , It is considered that this operator is suitable for PTRC, Then the optimizer will introduce PTRC.

PTRC How to determine the pairing operator and accelerate ？

Here we introduce a new concept ：

Parameterized repeat scan

, It means that the scanning operator scans the operator according to different parameters . such as Nested Loop Join, For every piece of data scanned on the surface , The internal table will be based on JOIN Condition to scan , So for the inner table, it's one time “ Parameterized repeat scan ”. Another example is related sub query , For each scan of the parent query, the sub query will be called to execute according to the result set of the parent query , Then return the result set of the subquery .

PTRC How it works ？

As mentioned earlier ,PTRC Is the intermediate result set of the cache operator , So it's also like other cache equally , Taking data to key ,value Cache to cache in , adopt key To hit , obtain value. that PTRC Correlation key and value How to get ？

Now let's Correlated subquery As an example to do a simple analysis , The query statement is as follows ：

SELECT *

FROM t1

WHERE t1.a IN (SELECT a

 FROM t2,

 t3

 WHERE t2.b = t1.b

 AND t2. c > t3.d); 

The above figure shows the subquery usage EXISTS Flow chart of policy implementation . It can be seen that ： For data sheets t1 Each piece of data in , Will drive subquery execution , Until the data sheet t1 All records in the end of the cycle . For data sheets t1 Corresponding to each record in t1.a, You need to rescan the subquery according to the column value , Then determine the return value of the sub query .

We go through EXPLAIN To compare the introduction of PTRC The difference between the implementation plan before and after ：

EXPLAIN format=tree

SELECT *

FROM t1

WHERE t1.a IN (SELECT a

 FROM t2,

 t3

 WHERE t2.b = t1.b

 AND t2. c > t3.d);
-> Filter: <in_optimizer>(t1.a,<exists>(select #2)) (cost=0.35 rows=1)

-> Table scan on t1 (cost=0.35 rows=1)

-> Select #2 (subquery in condition; dependent)

-> Result cache : cache keys(t1.a, t1.b)

-> Limit: 1 row(s) (cost=0.80 rows=1)

-> Filter: (t2.c > t3.d) (cost=0.80 rows=1)

-> Inner hash join (no condition) (cost=0.80 rows=1)

-> Table scan on t3 (cost=0.35 rows=2)

-> Hash

-> Filter: ((t2.b = t1.b) and (<cache>(t1.a) = t2.a)) (cost=0.35 rows=1)

-> Table scan on t2 (cost=0.35 rows=1)

It can be seen that PTRC after , One more operator

Result cache（ The red part ）

, It shows that the current subquery of this operator introduces PTRC, The implementation process after import is changed to ：

introduce PTRC after , For data sheets t1 Each piece of data in the corresponds to t1.a The column value , Priority view PTRC, If hit , Directly from PTRC Get result set in , Instead of executing subqueries . If you miss , You need to continue to execute the subquery in the same way , The results of subquery execution will be stored in PTRC in . If the next time the same column value drives the execution of the subquery , Can be directly from PTRC obtain .

How the optimizer chooses PTRC？

The optimizer is selecting for the operator PTRC Will rely on

Cost estimation

, Mainly to see

shooting

（ shooting  =  Number of rows with different key values / Total rows of key values ）,  If the hit rate is greater than rds_partial_result_cache_cost_threshold（ Refer to the following system variables for specific meanings ） The minimum cost of variable definition ,PTRC Will be chosen , On the contrary, they will not be selected . Have you chosen PTRC, Can pass Explain format=tree perhaps Explain analyze To observe the actual hit . Let's illustrate with an example ：

EXPLAIN analyze

SELECT *

FROM t1

WHERE t1.a IN (SELECT a

 FROM t2,

 t3

 WHERE t2.b = t1.b

 AND t2. c > t3.d);
-> Filter: <in_optimizer>(t1.a,<exists>(select #2)) (cost=0.35 rows=1) (actual time=3800.595..3800.595 rows=0 loops=1)

-> Table scan on t1 (cost=0.35 rows=1) (actual time=0.064..0.093 rows=1 loops=1)

-> Select #2 (subquery in condition; dependent)

-> Result cache : cache keys(t1.a, t1.b) (Cache Hits: 0, Cache Misses：1, Cache Evictions: 0, Cache Overflows: 0, Memory Usage: 40960 ) (actual time=0.115..0.115 rows=0 loops=1)

-> Limit: 1 row(s) (cost=0.80 rows=1) (actual time=0.094..0.094 rows=0 loops=1)

-> Filter: (t2.c > t3.d) (cost=0.80 rows=1) (actual time=0.093..0.093 rows=0 loops=1)

-> Inner hash join (no condition) (cost=0.80 rows=1) (actual time=0.092..0.092 rows=0 loops=1)

-> Table scan on t3 (cost=0.35 rows=2) (never executed)

-> Hash

-> Filter: ((t2.b = t1.b) and (<cache>(t1.a) = t1.a)) (cost=0.35 rows=1) (actual time=0.039..0.039 rows=0 loops=1)

-> Table scan on t2 (cost=0.35 rows=1) (actual time=0.038..0.038 rows=0 loops=1)

from Result cache You can see behind this operator ：

• Cache Hits: 0, Indicates that the number of hits is 0

• Cache Misses：1, Indicates that the number of misses is 1

• Cache Evictions: 0, Said the use of LRU Number of records eliminated

• Cache Overflows: 0, Memory overflow The number of times

• Memory Usage： 40960, Indicates the amount of memory used by the current query

Because the optimizer uses cost estimates to calculate whether to use PTRC, If the estimate is wrong ,PTRC There are still extra costs , For example, create your own data structures , And copies of records . To ensure the query performance as much as possible ,PTRC To take the

Dynamic feedback

The way to view PTRC Whether to continue to use in the actual implementation process .PTRC It will automatically determine whether the hit rate is suitable for retention PTRC, The optimizer depends on the number of misses , every other rds_partial_result_cache_hit_ratio_frequency Will check if the hit rate is lower than rds_partial_result_cache_min_hit_ratio. If below this value , The optimizer will automatically disable further use PTRC.

How the optimizer limits PTRC Memory usage

Because a single query can have multiple PTRC operator , Each operator uses memory to store cached data , So control PTRC Memory usage is very necessary , To prevent memory OOM.

Through system variables rds_partial_result_cache_max_mem_size To define all... Used by each query PTRC The maximum memory used by the operator . If PTRC The total amount of memory used exceeds this value , The optimizer will be based on LRU Algorithm to eliminate . If you pass LRU The algorithm can find records that are suitable for the size of the current stored records for elimination , The current record can be cached , Otherwise, the current record will not be cached .

PTRC How to configure ？

First ,PTRC On by default , Can pass Optimizer_switch Medium partial_result_cache Options change settings ： Set to ON, Enable PTRC, Otherwise, close . Through the... In the following table 4 System variables , Yes PTRC Make specific settings .

PTRC Performance testing

Here's how we use TPCH Of Q17 To test whether it is enabled under different data volumes PTRC Performance changes before and after .

because PTRC It's a cache, Therefore, the higher the hit rate, the higher the performance . Of course, if PTRC The higher the execution cost of related operators , that PTRC The higher the performance improvement .

MariaDB Of subquery cache It is a feature introduced to accelerate the correlation subquery of repeated scan operator , We refer to MariaDB Of subquery cache The test sample , Also based on dbt-3 scale 1  Data sets , test PTRC about correlated subquery Acceleration effect of .

because MySQL and MariaDB Index creation is different , Execution time and MariaDB Different , Here we only need to focus on the relative time . The test results are shown in the following table ：

It can be seen that ： The hit rate of the last row in the above table is 0 Under the circumstances ,PTRC Default check if miss 了 200 After , Will trigger the check hit rate , I found that the hit rate was too low , therefore PTRC Automatic failure , So you can see miss There are only 200 strip .

MariaDB For test samples and results, please refer to ：

https://mariadb.com/kb/en/subquery-cache/#performance-impact

.

PTRC It can be adapted to parameterized repeated scanning , As long as the hit rate is enough , To speed up execution . For a variety of operators in the query, including correlation sub , Nested Loop Join, Semijoin, Antijoin Both have accelerating effects .PTRC It's officially online , Welcome to use .

Click to follow , The first time to learn about Huawei's new cloud technology ~