Principle of universal gbase high availability synchronization tool in Nanjing University

Data synchronization is an essential role in distributed databases . It's used by Nantah gbse8a Database products , As a mature market product , Its data synchronization function is also very perfect , Whether in terms of function or performance , Are impeccable .

Here is a brief explanation gbase8a The principle of product data synchronization .

As we all know , Data synchronization is both functional and performance . That is, it should be fast and accurate .

If the synchronized data is wrong or lost, it is unintentionally a failure , Similarly, if the data synchronization is accurate , But every time I have to toss for a long time , This is also unbearable .

Get down to business ,gbase8a The data synchronization of the product adopts the working mode of pipeline , combination client-server The model works , Ensure concurrent processing , Ensure the accuracy of data verification during synchronization .

But piecemeal data parallelization

The above figure shows the data of the same table , Parallelization of different data slices . If the data of a table has 1g, It is divided into 3 Slice to 3 Each node , After parallelization , Time can be changed into the original 1/3

Multi slice data parallelization

  Thread processing piple line Pattern

 piple line Mode can make full use of the processing power of multithreading , For example, some threads read data , Some threads verify the read data , Some threads write the correct data or send the correct data .

Because each thread and the magnitude of calculation and processing may be different , such as check Faster ,read slower , This will cause some threads to be busy and others to be idle

Then you can use different number of threads according to the business volume

piple+multi pthd Pattern

 work -group Pattern ： In this mode, state machine management is required , But this can do the most with the least number of threads , But the disadvantage is that the thread context will switch frequently

 

The above is the current gbase The specific principle of synchronization . It makes full use of the idea of concurrency and multi-core .

Of course, it can be optimized later , such as perthd Pattern , Lockless queue mode ,RCU Queue processing . Or introduce a collaboration process for processing .