Data Lake: Data Integration Tool DataX

Series of topics: Data Lake Series Articles

---

1. What is DataX

DataX is the open source version of Alibaba Cloud DataWorks data integration, which is mainly used to realize offline synchronization between data.DataX is committed to realizing stable and efficient data synchronization between various heterogeneous data sources (ie different databases) including relational databases (MySQL, Oracle, etc.), HDFS, Hive, ODPS, HBase, FTP, etc.

In order to solve the synchronization problem of heterogeneous data sources, DataX turns the complex mesh synchronization link into a star data link, and DataX is responsible for connecting various data sources as an intermediate transmission carrier;

When you need to access a new data source, you only need to connect this data source to DataX, and you can synchronize with the existing data source as seamless data.

2. DataX3.0 framework design

DataX adopts the Framework + Plugin architecture, which abstracts the reading and writing of data sources as Reader/Writer plugins and incorporates them into the entire synchronization framework.

Reader (Acquisition Module)

Responsible for collecting data from the data source and sending the data to Framework.

Writer

Responsible for continuously fetching data from the Framework and writing the data to the destination.

Framework (Intermediary)

Responsible for connecting Reader and Writer as a data transmission channel for both, and handling core technical issues such as buffering, flow control, concurrency, and data conversion.

3.DataX3.0 Core Architecture

DataX completes a single data synchronization job, called Job. After DataX receives a Job, it will start a process to complete the entire job synchronization process.The DataX Job module is the central management node of a single job, and undertakes functions such as data cleaning, subtask segmentation, and TaskGroup management.

After the DataX Job is started, it will divide the Job into multiple small Tasks (subtasks) according to the segmentation strategies of different sources to facilitate concurrent execution.

Then the DataX Job will call the Scheduler module to reassemble the split tasks into a TaskGroup (task group) according to the configured number of concurrent tasks

Each Task is started by the TaskGroup. After the Task is started, the Reader --> Channel --> Writer thread will be started fixedly to complete the task synchronization.

After the DataX job is started, the Job will monitor the TaskGroup. After all the TaskGroups are completed, the Job will exit successfully (the value is not 0 when it exits abnormally)

DataX scheduling process:

1. First, the DataX Job module will be divided into several tasks according to the sub-database and sub-table;
2. Then calculate how many TaskGroups need to be allocated according to the number of concurrent user configuration: calculation process: Task / Channel = TaskGroup,
3. Finally, the TaskGroup runs the Task (task) according to the allocated concurrency

4. References

[01]https://www.jb51.net/article/241637.htm

---