Tidb 6.0: making Tso more efficient tidb Book rush

The author of this article ：h5n1,TiDB lovers , At present, he works in China Unicom software research institute ,asktug Home page

1. Preface

TiDB As a distributed database , Computing node tidb server And storage nodes tikv/tiflash server It has almost linear expansion ability , When resources are insufficient, you can directly expand the capacity online . But as a whole cluster of brains PD Node because only leader Provide services , It is not possible to increase processing power by extending nodes like other components .

at present TSO The main problem of distribution ：

1. TSO Distribution by PD Leader Nodes provide , A large number of requests will lead to Leader node CPU Increased utilization , Affect transaction latency .

2. PD Follower The node is basically idle , The utilization rate of system resources is low .

3. TiDB Cross data center access PD Leader when , Delays between data centers lead to increased transaction delays .

For promotion TSO The processing performance is targeted at some scenarios TiDB Introduced TSO Follower Proxy、RC Read TSO Optimize 、Local TSO Other characteristics , By extending the PD Processing capacity and reduction TSO How to request , Improve overall throughput , Reduce transaction latency .

2. TSO

TSO Is a monotonically increasing timestamp , from PD leader Distribute .TiDB At the beginning of the transaction, it gets TSO As start_ts、 Get... When submitting TSO As commit_ts, rely on TSO Implementation of transactions MVCC.TSO by 64 Bit integer value , It consists of physical part and logical part , high 48 The bit for the physical part is unixtime Time in milliseconds , low 18 The bit logic part is a numeric counter , Theoretically, it can produce 262144000( namely 2 ^ 18 * 1000) individual TSO.

To guarantee performance PD It does not generate one for each request TSO, Instead, an allocable time window will be requested in advance , The time window is the current time and the current time +3 Seconds later TSO, And keep it in etcd Inside , Then you can assign... From the window TSO. The update time window will be triggered every certain time . When PD Restart or leader After switching, it will start from etcd Get the maximum saved TSO Assigned starting , In order to make sure TSO Continuous increment of .

3. Follower Proxy

By default TSO Request by the PD leader Handle ,TiDB Through internal PD Client towards PD leader Send request to get TSO,PD client Requests received will not be sent immediately to PD leader , All requests received at the same time are packaged and sent to PD leader, Then from PD leader Return to batch TSO. Because only leader Provide services ,tidb server When the quantity is large, there will be more PD Client and PD Leader Establishing a connection , Causes switching to handle connection requests CPU Consumption is high . meanwhile follower Nodes only pass through raft Synchronize data and provide election and other functions , Basically idle .

stay 5.3.0 Version to introduce TSO Follower Proxy function , When on tidb Of PD Client Will randomly choose one PD node ( Include leader and follower ) send out TSO request ,PD Follower As a proxy service, it will receive a batch of requests by default PD Client Handle TSO Package and send to leader Handle , To further reduce PD Client and PD Leader The number of interactions and PD leader Connection number , To reduce leader Of CPU load .

By setting global variables tidb_enable_tso_follower_proxy by true Can be opened PD follower Of TSO Agent function , This function is applicable to tidb server A large number of concurrent requests ,PD leader Due to high pressure TSO Reach by request CPU bottleneck , Lead to TSO RPC Scenarios with high request latency .

4. RC Read TSO Optimize

Read-Commited The isolation level needs to be in pessimistic transaction mode , In pessimistic transactions, every SQL The execution will start from PD obtain TSO (for_update_ts) For consistent reading 、 Conflict detection, etc , Every SQL Equivalent to one Snapshot-Isolation Of ’ Small business ’, Compared with the optimistic transaction model, the pessimistic transaction produces TSO More requests , During the whole transaction, if we can reduce the number of transactions without destroying the consistency and isolation of transactions tso Number of requests , Can reduce PD Load and transaction latency , To improve overall performance .

6.0 It's right in the version RC In the transaction SELECT sentence TSO The request is optimized , Use an optimistic approach to get TSO , Get the latest version only when you encounter a new version TSO Reading data , By reducing read operations from PD obtain TSO Number of requests , This reduces query latency , Improve the performance of scenarios with small read-write conflicts QPS. This feature is provided by tidb_rc_read_check_ts Variable control , The default is off, Turn on this function and set it to on that will do .

After optimization select The basic process of statement processing is as follows ：

1. Select Statement execution does not start from PD obtain TSO As for_update_ts, Instead, use the last valid TSO As for_update_ts( That is to say read_ts). If it is the first statement in the transaction, it is start_ts, Otherwise it's the last SQL Of for_update_ts.

2. Build an execution plan and execute , Send to tikv Data read request for (pointget、coprocessor) Will take it RcReadCheckTS sign .

3. The data read request uses the previously obtained read_ts Do consistent reading , And return the data to tidb server.

4. TiKV It will check whether the returned data has an updated version , If there is an updated version, return WriteConflict error , Otherwise, the execution will normally end after the data is returned .

5. If at this time tidb Not yet client Send data from PD Get the latest TSO As for_update_ts Re execute the entire query , Otherwise, you will ask client Returns an error .

From the above process, we can see that when a new version is encountered, it will lead to tidb server Use the normal process to retrieve TSO And execution SQL, In case of read-write conflict, it will reduce performance and prolong transaction execution time . If some data has been returned client It will lead to an error SQL Execution failure , Although by increasing tidb_init_chunk_size Variable size delay tikv Return data time , It can reduce the occurrence of some of the above errors , But it is still not a fundamental solution .

5. Local TSO

In a multi data center scenario PD leader Located in a data center , Delays between data centers can cause TSO Request delay increases , If it can be done in the data center TSO Requests and allocations can be greatly reduced TSO Request delay . Based on this tidb Introduced Local TSO ( Experimental function ),PD Middle design 2 individual TSO allocator role ：local tso allocator and global tso allocator, The corresponding transactions are also divided into local transactions local transaction And global transactions global transaction Two kinds of .

• Local TSO

When passed enable-local-tso After being enabled, the data in the data center PD A node will be selected as local tso allocator Used to allocate TSO, This node acts as local tso The distribution of leader role ( PD The role is still follower ). When the data of transaction operation only involves the data of this data center , It is judged as a local transaction , To the local tso allocator apply local tso.

Each data center assigns its own local tso, They are independent of each other , To avoid assigning the same data to different data centers TSO,PD Will be set local tso The lower bits of the logical time in the are suffixed , Different data centers use different values , At the same time, this information will be persistently recorded to PD Inside .

• Global TSO

When the data of transaction operation involves other data centers, it is a global transaction , At this time, we need to ask PD leader apply global tso, PD leader As global tso allocator. When not enabled local-tso When the function , Still follow the original logic in all data TSO Request by the PD leader Responsible for handling .

To guarantee local tso and global tso The linear growth of ,global tso allocator and local tso allocator Will be carried out in max_tso Sync ：

1. Global tso allocator Collect all local tso allocator Maximum local tso.

2. From all local tso Choose one of the largest local tso As max_tso Send it to local allocator.

3. If max_tso Bigger than your own is newer TSO by max_tso, Otherwise, success will be returned directly .

Local tso It is necessary to consider that different data centers deal with different businesses , At the same time, we should combine PlacementRules in SQL Distribute tables by data center according to business rules , At the same time, you can set txn_scope Variable is local/global It is used for human control and acquisition global tso still local tso.

Basic configuration steps ( Currently not supported Local TSO Back to Global TSO Pattern )：

1. PD、TiKV、TiDB server Both need to be set according to the actual deployment label, To ensure high availability of each DC Of PD Quantity shall >1.

1. Open library or table level Placement Rules in SQL, Dispatch according to region and business relationship .

CREATE PLACEMENT POLICY dc1_leader LEADER_CONSTRAINTS="DC1" FOLLOWER_CONSTRAINTS="DC1,DC2,DC3" FOLLOWERS=2; Alter table new_order PARTITION p0 PLACEMENT POLICY dc1_leaders;

1. Set up PD Parameters enable-local-tso=on Use tiup reload restart PD Turn on Local TSO function . After enabling, you can use pd-ctl -u pd_ip:pd_port member in tso_allocator_leaders Item content view each center's local tso allocator leader.

6. test

6.1 Test environment

6.2 TSO Follower Proxy

stay 1024 Thread sysbench Yes 6 Zhang 1 The billion record table is opening TSO Follower Proxy Front and rear TPS And the average delay is as follows ：

Testing period TSO Follower Proxy Closing and opening CPU utilization ：

6.3 RC Read TSO

Use tiup-bench Test the startup under different threads tidb_rc_read_check_ts Front and rear TPCC, You can see that after the function is enabled, the TPCC There is a certain improvement , But as the number of threads increases, conflicts increase TPCC There is a drop .

adopt TiDB –> PD Client –> PD Client CMD OPS Monitoring can see 256 Start under the thread tidb_rc_read_check_ts after PD client Waiting in the TSO The frequency of .

6.4 Local TSO

Local TSO As an experimental function, it needs to be improved ,TPCC In the test, when this function is enabled, a large number of repeated primary key errors occur .

7. summary

For promotion TSO Scalability and efficiency ,TiDB A lot of optimization work has been done , But these optimizations have certain scenarios , It needs to be considered in combination with the business and actual situation , Otherwise, blind opening may cause QPS Reduce 、 Delayed increase ：

about Follower TSO Proxy Suitable for as a result of PD Leader CPU It's busy TSO Get delayed scenarios , Through the open Follower Proxy And then it goes down leader The pressure of the .

RC Read TSO Optimization is suitable for scenarios with more reading and less writing , If the data conflict is serious, the performance will be degraded .

Local TSO As TiDB The distributed timing scheme can theoretically solve the problem caused by the delay between data centers TSO Delay , However, there are still some problems with the experimental function .