Tidb v6.0.0 (DMR): initial test of cache table

The author of this article ： Cheerleading ,TiDB Old powder , Currently working in JD logistics , Senior users in the community ,asktug Home page

One 、 background

In general use TiDB The size of a single table is more than ten million, which is the best in the business , But in real business, there are always small tables . For example, the configuration table has few write requests , The performance requirements for read requests are higher .TiDB As a distributed database , The load of large tables can easily be distributed to multiple machines by using the distributed feature , But when the amount of data in the table is small , When visits are particularly frequent , Data is usually concentrated in TiKV One of the Region On , Form a reading hotspot , More likely to cause performance bottlenecks .

TiDB v6.0.0(DMR) This version introduces the function of caching tables , When I first saw this word, I thought of MySQL Memory table of .MySQL The table structure of the memory table is created on the disk , Data is stored in memory . The disadvantages of memory tables are obvious . When MySQL When it starts , Both tables and data exist , When MySQL After restart , Table structure exists , The data disappears .TiDB The cache table of does not have this problem . from asktug In the forum, I saw that many small partners are looking forward to the performance of the cache table , I am also looking forward to its performance , So actually look at the performance of the cache table in the test environment .

Two 、 Usage scenarios for caching tables

The following sections are from official documents , For details, see Cache table

TiDB The cache table function is applicable to tables with the following characteristics ：

The amount of data in the table is not large
Read only tables , Or very little modification
Tables are accessed frequently , Expect better read performance

As for the first point, the official document mentions that the size of the cache table is limited to all including indexes key-value The total size of the record cannot exceed 64 MB. The actual test uses Sysbench Generate the table structure in the following table from 20w Up to 30w Cannot convert a normal table to a cached table when the data volume is , Therefore, the actual use of cache tables in the production environment should not exceed hundreds of thousands of levels of data at most . About the performance of the cache table in terms of containing read and write operations , Tests were conducted using a variety of different read / write request ratios , Compared with ordinary meters, it does not achieve better performance . This is because the consistency of the read data , After performing the modification operation on the cache table , Write operations will be blocked during lease time , The longest possible occurrence tidb_table_cache_lease Long wait for variable value , It can lead to QPS Reduce . Therefore, cache tables are more suitable for read-only tables , Or scenes that are rarely modified .

The cache table transfers the data of the entire table from TiKV Load into TiDB Server in , You can query without accessing TiKV Directly from TiDB Server Read from the cache of , Save disk IO And network bandwidth . When using a normal table query , The more data returned, the less efficient the index may be , Until the cost of full table scanning approaches, the optimizer may directly select full table scanning . The data of the cache table itself is TiDB Server The memory of the , Disk can be avoided IO, Therefore, the query efficiency will be higher . Take the configuration table as an example , When the business restarts , All business connections are loaded together with the configuration , This will cause high database read latency . If a cache table is used , Read requests can read data directly from memory , It can effectively reduce the read delay . In the financial scenario , Business usually involves both order table and exchange rate table . Exchange rate tables are usually small , The table structure rarely changes, so there is almost no DDL, Plus, it is only updated once a day , It is also very suitable for using cache tables . Other business scenarios, such as Bank branch or outlet information table , Cities in the logistics industry 、 Warehouse No. warehouse No. table , Areas of e-commerce industry 、 Category related dictionaries, etc , Tables with few new entries are typical usage scenarios for cache tables .

3、 ... and 、 Test environment

1. Hardware configuration and cluster topology planning

Use 2 Virtual machine , The hardware configuration is 4C 16G 100G Ordinary SSD Hard disk .

Role	Host	Ports
alertmanager	10.0.0.1	9093/9094
grafana	10.0.0.1	3000
pd	10.0.0.1	2379/2380
pd	10.0.0.2	2379/2380
pd	10.0.0.1	3379/3380
prometheus	10.0.0.1	9090/12020
tidb	10.0.0.1	4000/10080
tidb	10.0.0.2	4000/10080
tikv	10.0.0.1	20162/20182
tikv	10.0.0.1	20160/20180
tikv	10.0.0.2	20161/20181

2. software configuration

Software name	Software usage	edition
CentOS	operating system	7.6
TiDB colony	Open source distributed NewSQL database	v6.0.0 DMR
Sysbench	Pressure test tools	1.0.20

3. Parameter configuration

server_configs:
tidb:
log.slow-threshold: 300
new_collations_enabled_on_first_bootstrap: true

tikv:
readpool.coprocessor.use-unified-pool: true
readpool.storage.use-unified-pool: false
pd:
replication.enable-placement-rules: true
    replication.location-labels:
    - host

Due to limited hardware conditions , Only 2 A cluster with mixed deployment of virtual machines with common performance （ In fact, it is similar to single machine deployment ）. stand-alone CPU Fewer cores and TiDB Server There is no load balancing, so the concurrency cannot be adjusted too high . The following tests use a TiDB Server Conduct pressure measurement at the nodes , Therefore, there is no need to pay special attention to the test data of this test , It may be different from other test results , Does not represent best performance practices and deployment , Test results are for reference only .

Four 、 Performance testing

Sysbench Generated table structure

CREATE TABLE sbtest1 (
id int(11) NOT NULL AUTO_INCREMENT,
k int(11) NOT NULL DEFAULT '0',
c char(120) NOT NULL DEFAULT '',
pad char(60) NOT NULL DEFAULT '',
PRIMARY KEY (id),
KEY k_1 (k)
) ENGINE = InnoDB CHARSET = utf8mb4 COLLATE = utf8mb4_bin AUTO_INCREMENT = 1

Read performance test

Test the main parameters oltp_point_select Primary key query test （ Check , The condition is unique index column ）

The main SQL sentence ：

SELECT c FROM sbtest1 WHERE id=?

select_random_points Random multiple queries （ Of primary key columns selete in operation ）

The main SQL sentence ：

SELECT id, k, c, pad FROM sbtest1 WHERE k IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)

select_random_ranges Random range query （ Of primary key columns selete between and operation ）

The main SQL sentence ：

SELECT count(k) FROM sbtest1 WHERE k BETWEEN ? AND ? OR k BETWEEN ? AND ? OR k BETWEEN ? AND ? OR k BETWEEN ? AND ? OR k BETWEEN ? AND ? OR k BETWEEN ? AND ? OR k BETWEEN ? AND ? OR k BETWEEN ? AND ? OR k BETWEEN ? AND ? OR k BETWEEN ? AND ?

oltp_read_only read-only operation （ Including aggregation 、 Go ahead and wait ）

The main SQL sentence ：

SELECT c FROM sbtest1 WHERE id=?

SELECT SUM(k) FROM sbtest1 WHERE id BETWEEN ? AND ?

SELECT c FROM sbtest1 WHERE id BETWEEN ? AND ? ORDER BY c

SELECT DISTINCT c FROM sbtest1 WHERE id BETWEEN ? AND ? ORDER BY c

Sysbench Test command example

sysbench --mysql-host=10.0.0.1 --mysql-port=4000 --mysql-db=sbtest --mysql-user=root --time=600 \
--threads=8 --report-interval=10 --db-driver=mysql oltp_point_select --tables=1 --table-size=5000 run

sysbench --mysql-host=10.0.0.1 --mysql-port=4000 --mysql-db=sbtest --mysql-user=root --time=600 \
--threads=8 --report-interval=10 --db-driver=mysql oltp_read_only --tables=1 --table-size=5000 run

sysbench --mysql-host=10.0.0.1 --mysql-port=4000 --mysql-db=sbtest --mysql-user=root --time=600 \
--threads=8 --report-interval=10 --db-driver=mysql select_random_points --tables=1 --table-size=5000 run

sysbench --mysql-host=10.0.0.1 --mysql-port=4000 --mysql-db=sbtest --mysql-user=root --time=600 \
--threads=8 --report-interval=10 --db-driver=mysql select_random_ranges --tables=1 --table-size=5000 run

One 、 Use regular tables

1. Single table data volume 5000, test QPS

threads/request type	oltp_point_select	oltp_read_only	select_random_points	select_random_ranges
8	2214	1985	3190	2263
16	3199	2414	3412	2491
32	4454	2867	3898	2743
64	5792	3712	4321	2981
128	7639	4964	4474	2965

2. Single table data volume 50000, test QPS

threads/request type	oltp_point_select	oltp_read_only	select_random_points	select_random_ranges
8	4874	2808	2841	2207
16	5042	3429	3172	2448
32	6754	4290	3405	2651
64	8989	5282	3831	2818
128	12565	6470	3996	2811

Two 、 Use cache table

1. Single table data volume 5000, test QPS

threads/request type	oltp_point_select	oltp_read_only	select_random_points	select_random_ranges
8	15780	10811	5666	2716
16	23296	11399	6417	2948
32	28038	11313	6907	3050
64	32924	11377	7217	3200
128	33962	11413	7199	3232

2. Single table data volume 50000, test QPS

threads/request type	oltp_point_select	oltp_read_only	select_random_points	select_random_ranges
8	15910	16540	5359	2646
16	21945	17022	5999	2915
32	25614	17356	6355	3065
64	31782	17410	6690	3088
128	35009	17584	6713	3161

3、 ... and 、 Performance comparison

Read write hybrid performance test

Test the main scenario parameters oltp_read_write Indicates mixed read / write .

point_selects（ The number of queries per transaction )

delete_inserts（ Insert... Into each transaction / Delete the number of combinations ）

The main SQL sentence ：

INSERT INTO sbtest1 (id, k, c, pad) VALUES (?, ?, ?, ?)

DELETE FROM sbtest1 WHERE id=?

SELECT c FROM sbtest1 WHERE id=?

This test passes the number of request types in a single transaction --delete_inserts Fixed for 10 And adjust --point_selects Parameter to simulate the performance difference under different read / write ratios , Other request parameters use default values , Refer to the following for specific commands Sysbench Test command example .

Sysbench Test command example

sysbench --mysql-host=10.0.0.1 --mysql-port=4000 --mysql-db=sbtest --mysql-user=root --time=600 --threads=8 --report-interval=10 --db-driver=mysql oltp_read_write --tables=1 --table-size=5000 --point_selects=10 --non_index_updates=0 --delete_inserts=10 --index_updates=0 run

One . Use regular tables

1. Single table data volume 5000, test QPS

threads/--point_selects	10	40	160	640
8	869	2289	3852	5090
16	1014	2139	4354	6094
32	1075	2205	5089	6944
64	605	1861	5160	8395
128	877	2127	4332	9257

2. Single table data volume 50000, test QPS

threads/--point_selects	10	40	160	640
8	1107	2144	3312	4439
16	1108	2103	3738	5702
32	1055	2228	4325	6770
64	1062	1397	5367	8209
128	981	1838	7235	17472

Two 、 Use cache table

1. Single table data volume 5000, test QPS

threads/--point_selects	10	40	160	640
8	711	1322	2123	2787
16	361	665	1274	2870
32	400	627	1394	2997
64	323	804	1853	4100
128	372	680	1847	4704

2. Single table data volume 50000, test QPS

threads/--point_selects	10	40	160	640
8	974	2726	3716	1804
16	787	1366	1736	2176
32	673	1231	2338	4627
64	572	1384	3120	7755
128	557	1104	2907	7486

3、 ... and 、 Performance comparison

5、 ... and 、 Problems encountered

Try to 30w An error occurs when the data table is changed to a cache table ERROR 8242 (HY000): 'table too large' is unsupported on cache tables.

at present TiDB The size of each cache table is limited to 64 MB, Therefore, too large tables cannot be cached in memory . in addition , The cache table cannot perform normal DDL sentence . To execute... On the cache table DDL sentence , You need to use it first ALTER TABLE xxx NOCACHE Statement to remove the cache attribute , After setting the cache table back to the normal table , To perform other DDL sentence .

The performance of the cache table is unstable during the test , Sometimes the read performance of cached tables is worse than that of ordinary tables , Use trace sentence （TRACE SELECT * FROM sbtest1;） After checking, I found that... Appears in the returned results regionRequest.SendReqCtx, explain TiDB Not all data has been loaded into memory , Multiple attempts failed to load . hold tidb_table_cache_lease Adjusted for 10 This problem did not occur after .

stay asktug China has raised this question to the R & D leader and got the answer . according to github.com/pingcap/tid… Description in , When the machine is heavily loaded ,load table need 3s above , But by default tidb_table_cache_lease yes 3s, Indicates that the loaded data is immediately obsolete , So you need to reload , And the process repeats forever . It leads to a great waste of CPU Resources and reduced QPS. At present, we can put tidb_table_cache_lease To increase the value of , The problem lies in master Has been resolved in the branch , Subsequent versions should not appear .

According to the test results , The performance of the cache table is poor when writes are frequent . In tests that include write requests , Compared with ordinary tables, the performance of cached tables is almost significantly reduced .

stay lease Before the expiration date , Cannot modify data . To ensure data consistency , Modification must wait lease Be overdue , So there is a write delay . for example 1tidb_table_cache_lease1 by 10 when , There may be a large delay in writing . Therefore, it is not recommended to use cache tables for businesses with frequent writes or high write latency requirements .

6、 ... and 、 Test summary

Read performance

Single table 5000, The percentage of cache table improvement compared with normal table

threads/request type	oltp_point_select	oltp_read_only	select_random_points	select_random_ranges
8	612.73%	444.63%	77.61%	20.01%
16	628.22%	372.20%	88.01%	18.34%
32	529.50%	294.59%	77.19%	10.38%
64	468.43%	206.49%	67.02%	7.34%
128	344.58%	129.91%	60.90%	9.00%

Single table 50000, The percentage of cache table improvement compared with normal table

threads/request type	oltp_point_select	oltp_read_only	select_random_points	select_random_ranges
8	226.42%	489.03%	88.63%	19.89%
16	335.24%	396.41%	89.12%	19.07%
32	279.24%	304.56%	86.63%	15.61%
64	253.56%	229.60%	74.62%	9.58%
128	178.62%	171.77%	67.99%	12.45%
### Read write mix
Single table 5000, The percentage of cache table improvement compared with normal table （ Negative growth is in line with expectations ）

threads/--point_selects	10	40	160	640
8	-35.77%	-42.24%	-44.88%	-45.24%
16	-64.39%	-68.91%	-70.73	-52.90%
32	-62.79%	-71.56%	-72.60%	-56.84%
64	-46.61%	-42.44%	-64.08%	-50.05%
128	-57.58%	-68.03%	-57.36%	-49.18%

Single table 50000, The percentage of cache table improvement compared with normal table （ Negative growth is in line with expectations ）

threads/--point_selects	10	40	160	640
8	-12.01%	27.14%	12.19%	-59.36%
16	-28.97%	-35.04%	-53.55%	-61.83%
32	-36.20%	-44.74%	-45.94%	-31.65%
64	-46.13%	-0.93%	-41.86%	-5.53%
128	-43.21%	-39.93%	-59.82%	-57.15%

Results show , Compared with ordinary tables , Cache table in oltp_point_select、oltp_read_only、select_random_points、select_random_ranges The performance is greatly improved in several read-only scenarios , But it doesn't provide better performance in tests that include write requests . Its mechanism determines that the usage scenario is currently limited to read-only tables with a small amount of data , Or small tables that are rarely modified . Sum up , Although the current usage scenario of the cache table is relatively simple , However, it is a good function to solve the business pain points under the appropriate scenarios , We also look forward to higher stability and better performance in subsequent versions .