Based on the open source stream batch integrated data synchronization engine Chunjun data restore DDL parsing module actual combat sharing

Link to the original text ： Based on the open source stream batch integrated data synchronization engine ChunJun Data restore —DDL Share the actual combat of the analysis module

Courseware acquisition ： Official account ** “ Several stack Study Club ”, Backstage private message “ChunJun”** Get live courseware

Video playback ： Click here

ChunJun Open source project address ：github 丨 gitee Like our project, give us a STAR！STAR！！STAR！！！（ Important things are to be repeated for 3 times ）

One 、ChunJun-DDL Introduction to the parsing module

In the last issue, we introduced 《ChunJun Support heterogeneous data sources DDL Conversion and automatic execution 》, Among them the Calcite analysis DDL The actual combat content is briefly introduced , In this issue, we will share this part in detail . It's going on DDL Before actual demonstration , Let's first review data restore and DDL Parse the contents of the module .

1、 Introduction to data restore

ChunJun Support mysql oracle postgresql sqlserver And other data sources are synchronized in real time , But the data after synchronization is output in the form of log , On this basis, data restoration can make the changes of source data change correspondingly in the target table , contain DML as well as DDL The corresponding operations will be performed in the target table , Ensure that the source table and target table schema Agreement 、 The data is consistent .

stay ChunJun There are also two scenarios for data restore in ： Manual and automatic execution , The difference between them is ChunJun Can you parse Source End DDL Statement and convert to the downstream syntax corresponding statement , At the same time, the downstream will execute the corresponding DDL.

2、DDL Introduction to the parsing module

Because different data sources have different syntax , Execute in the heterogeneous data source scenario of data restore DDL Is a problem , How to integrate Source Of DDL Statement is converted to conform to Sink The grammatical DDL sentence , It has become a problem that puzzles many people ,ChunJun Also faced with such a dilemma , To solve the problem ,ChunJun Introduced DDL The parsing module solves this problem .

DDL The parsing module mainly solves 2 A question

• analysis Source Of SQL, Turn it into ChunJun Extracted public entity classes

• The public entity class is divided into Sink The syntax rule of the end is changed to the corresponding SQL

about SQL Convert this part , We added a DDL Convent Interface , This interface provides SQL Convert to public entity class and public entity class back to SQL The function of , The specific conversion will be implemented by the corresponding data source plug-ins according to their respective syntax .

DDL Data It is a top-level abstract class of public layer data that we extract , Because the syntax rules of each data source are different , So we need to extract a common layer , This common layer is used to convert data between different data sources .

To analyze SQL And turn it into public layer data , We introduced Calcite Conduct SQL Parsing , Different data sources will have corresponding SQL Parser , Through this parser SQL Parsing , take SQL The statement changes to the corresponding SQLNode, then SQLNode The information in is transferred to the middle tier .

In this way, we have solved the problem of DDL Grammatical inconsistencies .

Two 、ChunJun-DDL Analysis module practice

In the last chapter, we introduced some basic concepts , Next, let's show you how to operate .

- First step

stay ChunJun-SQL module Next , perform mvn clean compile -DskipTest,maven Will be packaged based on Calcite Implemented parser , Various data sources DDL The parsing module will generate the corresponding parser for SQL Parsing .

- The second step

The generated parser supports parsed SQL Syntax can see the corresponding module test modular .

- Concrete demonstration

The actual demonstration process will not be reviewed on the official account , Community partners can go to B Check the live review video on the website .

B Station live review address ：

https://www.bilibili.com/video/BV1M3411w7S3?spm_id_from=333.999.0.0

3、 ... and 、Calcite stay ChunJun-DDL Practical application in

We introduce Calcite Conduct SQL Parsing , In the last chapter, we discussed ChunJun-DDL The analysis module is demonstrated in practice , Now let's share Calcite stay ChunJun-DDL Practical application in .

- First step

Mavne rely on , concrete ChunJun-DDL-MySQL Module Of pom To configure .

- The second step

Main Add the corresponding Codegen Folder , It is mainly composed of config.fmpp（Calcite Configuration file for ） and FTL The grammar file consists of .

- The third step

Write according to the data source syntax FTL file

Calcite Use Javacc As a grammar parser , And use Freemarker As a template engine , At compile time ,Freemarker The profile 、 Template syntax file 、 Attach the template file to generate the final syntax file as a whole , And pass JavaCC compile , formation Calcite The grammar file of .

- Concrete demonstration

The actual demonstration process will not be reviewed on the official account , Community partners can go to B Check the live review video on the website .

B Station live review address ：

https://www.bilibili.com/video/BV1M3411w7S3?spm_id_from=333.999.0.0

The source of the original ：VX official account “ Several stack Study Club ” Kangaroo cloud open source framework nail technology exchange group （30537511）, Students interested in big data open source projects are welcome to join us to exchange the latest technical information , Open source project library address ：https://github.com/DTStack