Overview of Yunxi database executor

as everyone knows , Before actuator execution , Need the support of the plan . Plans are divided into logical plans and physical plans . Their relationship is like we're going on a trip , Choosing which means of transportation is equivalent to logical planning , In this step, for example, after selecting an aircraft . Choosing which airline is equivalent to a physical plan . Last , When you really start to travel, it's equivalent to carrying out . Here's how SQL Basic architecture diagram of statement execution , From this, we can clearly see the execution process of optimizer and executor in the whole process .

Optimizer

Logical plan and physical plan are responsible for generating execution plan and index selection , Think of it as an optimizer . For example, executing such a statement , Execute the of two tables join：

Select * from t1 join t2 using(ID) where t1.c = 10 and t2.d = 20;

You can start with t1 Take it out c=10 Records of the ID, According to ID Related to t2, To determine t2 Inside d Is the value of 20, You can also start with t2 Take it out c=20 Records of the ID, According to ID Related to t1, To determine t2 Inside d Is the value of 10. The logic of these two execution methods is the same , But the execution efficiency is different , The optimizer can estimate the cost and decide to use the scheme . In distributed database , The physical plan can also be based on the data to be used span The node where the operator is located determines which node the operator is executed on , So as to realize distributed execution . Overview of the specific role of distributed related physical plans in distributed execution .

 

Distributed execution

The key idea of distributed execution is how to change from logical execution plan to physical execution plan , There are mainly two aspects involved here , One is the distributed processing of computing , One is distributed data processing .

Once the physical plan is generated , The system needs to split it up and distribute it to each node Running between . Every node Responsible for local dispatching processors and inputs.node You also need to be able to communicate with each other to put the output output router Connect to input. especially , Need one streaming interface To connect these components . To avoid extra synchronization costs , A flexible execution environment is needed to satisfy all of the above operations , So that different node In addition to the initial scheduling of the execution plan , Can be relatively independent to start the corresponding data processing work , Without being gateway Other choreographing effects of nodes .

In a cluster of databases Gateway node A scheduler will be created , It accepts a set of flow, Set input and output related information , To create local processor And start execution . stay node When processing input and output data , We need to be right about flow Take some control , Through this control , We can refuse request Some of the requests in .

Every Flow Represents a complete piece of execution across nodes in the entire physical plan , from processors and streams form , You can pull the data of the fragment 、 Data processing and final data output . As shown in the figure below ：

For cross node execution ,Gateway node First, we will serialize the right FlowSpec by SetupFlowRequest, And pass GRPC Send to the far end node, Distal node After receiving , It will be restored first Flow, And create what it contains processor And interactive use stream（TCP passageway ）, Complete the construction of the implementation framework , Then start the multi node calculation driven by the mesh joint points .Flow Through between box The cache pool is scheduled asynchronously , To realize the parallel execution of the whole distributed framework .

For local execution , It's parallel execution , Every processor,synchronizer and router Can be used as goroutine function , Between them by channel interconnection . these channel Channels can be buffered to synchronize producers and consumers .

To achieve distributed concurrent execution , The database introduces Router The concept of , about JOIN and AGGREGATOR According to the data distribution characteristics , Three ways of data redistribution are realized ,mirror_router、hash_router and range_router, By redistributing data processor The operator is internally split into two phases , The first stage is to process part of the data in the node where the data is located , After processing, the results , It will be redistributed according to the operator type , The second stage is aggregation , In this way, a single operator and multi node cooperative execution are realized .