Tidb 6.0: making Tso more efficient tidb Book rush

The author of this article

：h5n1,TiDB lovers , At present, he works in China Unicom software research institute ,asktug  Home page

1.  Preface

TiDB  As a distributed database , Computing node  tidb server  And storage nodes  tikv/tiflash server  It has almost linear expansion ability , When resources are insufficient, you can directly expand the capacity online . But as a whole cluster of brains  PD  Node because only  leader  Provide services , It is not possible to increase processing power by extending nodes like other components .

at present  TSO  The main problem of distribution ：

• TSO  Distribution by  PD Leader  Nodes provide , A large number of requests will lead to  Leader  node  CPU  Increased utilization , Affect transaction latency .

• PD Follower  The node is basically idle , The utilization rate of system resources is low .

• TiDB  Cross data center access  PD Leader  when , Delays between data centers lead to increased transaction delays .

For promotion  TSO  The processing performance is targeted at some scenarios  TiDB  Introduced  TSO Follower Proxy、RC Read TSO  Optimize 、Local TSO  Other characteristics , By extending the  PD  Processing capacity and reduction  TSO  How to request , Improve overall throughput , Reduce transaction latency .

2. TSO

TSO  Is a monotonically increasing timestamp , from  PD leader  Distribute .TiDB  At the beginning of the transaction, it gets  TSO  As  start_ts、 Get... When submitting  TSO  As  commit_ts, rely on  TSO  Implementation of transactions  MVCC.TSO  by  64  Bit integer value , It consists of physical part and logical part , high  48  The bit for the physical part is  unixtime  Time in milliseconds , low  18  The bit logic part is a numeric counter , Theoretically, it can produce  262144000( namely  2 ^ 18 * 1000) individual  TSO.

To guarantee performance  PD  It does not generate one for each request  TSO, Instead, an allocable time window will be requested in advance , The time window is the current time and the current time +3  Seconds later  TSO, And keep it in  etcd  Inside , Then you can assign... From the window  TSO. The update time window will be triggered every certain time . When  PD  Restart or  leader  After switching, it will start from  etcd  Get the maximum saved  TSO  Assigned starting , In order to make sure  TSO  Continuous increment of .

3. Follower Proxy

By default  TSO  Request by the  PD leader  Handle ,TiDB  Through internal  PD Client  towards  PD leader  Send request to get  TSO,PD client  Requests received will not be sent immediately to  PD leader , All requests received at the same time are packaged and sent to  PD leader, Then from  PD leader  Return to batch  TSO. Because only  leader  Provide services ,tidb server  When the quantity is large, there will be more  PD Client  and  PD Leader  Establishing a connection , Causes switching to handle connection requests  CPU  Consumption is high . meanwhile  follower  Nodes only pass through  raft  Synchronize data and provide election and other functions , Basically idle .

stay  5.3.0  Version to introduce  TSO Follower Proxy  function , When on  tidb  Of  PD Client  Will randomly choose one  PD  node ( Include  leader  and  follower ) send out  TSO  request ,PD Follower  As a proxy service, it will receive a batch of requests by default  PD Client  Handle  TSO  Package and send to  leader  Handle , To further reduce  PD Client  and  PD Leader  The number of interactions and  PD leader  Connection number , To reduce  leader  Of  CPU  load .

By setting global variables  tidb_enable_tso_follower_proxy  by  true  Can be opened  PD follower  Of  TSO  Agent function , This function is applicable to  tidb server  A large number of concurrent requests ,PD leader  Due to high pressure  TSO  Reach by request  CPU  bottleneck , Lead to  TSO RPC  Scenarios with high request latency .

4. RC Read TSO  Optimize

Read-Commited  The isolation level needs to be in pessimistic transaction mode , In pessimistic transactions, every  SQL  The execution will start from  PD  obtain  TSO (for_update_ts)  For consistent reading 、 Conflict detection, etc , Every  SQL  Equivalent to one  Snapshot-Isolation  Of ’ Small business ’, Compared with the optimistic transaction model, the pessimistic transaction produces  TSO  More requests , During the whole transaction, if we can reduce the number of transactions without destroying the consistency and isolation of transactions  tso  Number of requests , Can reduce  PD  Load and transaction latency , To improve overall performance .

6.0  It's right in the version  RC  In the transaction  SELECT  sentence  TSO  The request is optimized , Use an optimistic approach to get  TSO , Get the latest version only when you encounter a new version  TSO  Reading data , By reducing read operations from  PD  obtain  TSO  Number of requests , This reduces query latency , Improve the performance of scenarios with small read-write conflicts  QPS. This feature is provided by  tidb_rc_read_check_ts  Variable control , The default is  off, Turn on this function and set it to  on  that will do .

After optimization  select  The basic process of statement processing is as follows ：

• Select  Statement execution does not start from  PD  obtain  TSO  As  for_update_ts, Instead, use the last valid  TSO  As  for_update_ts( That is to say  read_ts). If it is the first statement in the transaction, it is  start_ts, Otherwise it's the last  SQL  Of  for_update_ts.

• Build an execution plan and execute , Send to  tikv  Data read request for (pointget、coprocessor) Will take it  RcReadCheckTS  sign .

• The data read request uses the previously obtained  read_ts  Do consistent reading , And return the data to  tidb server.

• TiKV  It will check whether the returned data has an updated version , If there is an updated version, return  WriteConflict  error , Otherwise, the execution will normally end after the data is returned .

• If at this time  tidb  Not yet  client  Send data from  PD  Get the latest  TSO  As  for_update_ts  Re execute the entire query , Otherwise, you will ask  client  Returns an error .

From the above process, we can see that when a new version is encountered, it will lead to  tidb server  Use the normal process to retrieve  TSO  And execution  SQL, In case of read-write conflict, it will reduce performance and prolong transaction execution time . If some data has been returned  client  It will lead to an error  SQL  Execution failure , Although by increasing  tidb_init_chunk_size  Variable size delay  tikv  Return data time , It can reduce the occurrence of some of the above errors , But it is still not a fundamental solution .

5. Local TSO

In a multi data center scenario  PD leader  Located in a data center , Delays between data centers can cause  TSO  Request delay increases , If it can be done in the data center  TSO  Requests and allocations can be greatly reduced  TSO  Request delay . Based on this  tidb  Introduced  Local TSO ( Experimental function ),PD  Middle design  2  individual  TSO allocator  role ：local tso allocator  and  global tso allocator, The corresponding transactions are also divided into local transactions  local transaction  And global transactions  global transaction  Two kinds of .

• Local TSO

When passed  enable-local-tso  After being enabled, the data in the data center  PD  A node will be selected as  local tso allocator  Used to allocate  TSO, This node acts as  local tso  The distribution of  leader  role ( PD  The role is still  follower ). When the data of transaction operation only involves the data of this data center , It is judged as a local transaction , To the local  tso allocator  apply  local tso.

Each data center assigns its own  local tso, They are independent of each other , To avoid assigning the same data to different data centers  TSO,PD  Will be set  local tso  The lower bits of the logical time in the are suffixed , Different data centers use different values , At the same time, this information will be persistently recorded to  PD  Inside .

• Global TSO

When the data of transaction operation involves other data centers, it is a global transaction , At this time, we need to ask  PD leader  apply  global tso, PD leader  As  global tso allocator. When not enabled  local-tso  When the function , Still follow the original logic in all data  TSO  Request by the  PD leader  Responsible for handling .

To guarantee  local tso  and  global tso  The linear growth of ,global tso allocator  and  local tso allocator  Will be carried out in  max_tso  Sync ：

• Global tso allocator  Collect all  local tso allocator  Maximum  local tso.

• From all  local tso  Choose one of the largest  local tso  As  max_tso  Send it to  local allocator.

• If  max_tso  Bigger than your own is newer  TSO  by  max_tso, Otherwise, success will be returned directly .

Local tso  It is necessary to consider that different data centers deal with different businesses , At the same time, we should combine  PlacementRules in SQL  Distribute tables by data center according to business rules , At the same time, you can set  txn_scope  Variable is  local/global  It is used for human control and acquisition  global tso  still  local tso.

Basic configuration steps  ( Currently not supported  Local TSO  Back to  Global TSO  Pattern )：

• PD、TiKV、TiDB server  Both need to be set according to the actual deployment  label, To ensure high availability of each  DC  Of  PD  Quantity shall >1.

• Open library or table level  Placement Rules in SQL, Dispatch according to region and business relationship .

CREATE PLACEMENT POLICY dc1_leader LEADER_CONSTRAINTS="DC1" FOLLOWER_CONSTRAINTS="DC1,DC2,DC3" FOLLOWERS=2; Alter table new_order PARTITION p0 PLACEMENT POLICY dc1_leaders;

• Set up  PD  Parameters  enable-local-tso=on  Use  tiup reload  restart  PD  Turn on  Local TSO  function . After enabling, you can use  pd-ctl -u pd_ip:pd_port member  in  tso_allocator_leaders  Item content view each center's  local tso allocator leader.

• 

6.  test

6.1  Test environment

6.2 TSO Follower Proxy

stay  1024  Thread  sysbench  Yes  6  Zhang  1  The billion record table is opening  TSO Follower Proxy  Front and rear  TPS  And the average delay is as follows ：

Testing period  TSO Follower Proxy  Closing and opening  CPU  utilization ：

6.3 RC Read TSO

Use  tiup-bench  Test the startup under different threads  tidb_rc_read_check_ts  Front and rear  TPCC, You can see that after the function is enabled, the  TPCC  There is a certain improvement , But as the number of threads increases, conflicts increase  TPCC  There is a drop .

adopt  TiDB –> PD Client –> PD Client CMD OPS  Monitoring can see  256  Start under the thread  tidb_rc_read_check_ts  after  PD client  Waiting in the  TSO  The frequency of .

6.4 Local TSO

Local TSO  As an experimental function, it needs to be improved ,TPCC  In the test, when this function is enabled, a large number of repeated primary key errors occur .

7.  summary

For promotion  TSO  Scalability and efficiency ,TiDB  A lot of optimization work has been done , But these optimizations have certain scenarios , It needs to be considered in combination with the business and actual situation , Otherwise, blind opening may cause  QPS  Reduce 、 Delayed increase ：

about  Follower TSO Proxy  Suitable for as a result of  PD Leader CPU  It's busy  TSO  Get delayed scenarios , Through the open  Follower Proxy  And then it goes down  leader  The pressure of the .

RC Read TSO  Optimization is suitable for scenarios with more reading and less writing , If the data conflict is serious, the performance will be degraded .

Local TSO  As  TiDB  The distributed timing scheme can theoretically solve the problem caused by the delay between data centers  TSO  Delay , However, there are still some problems with the experimental function .