Tdengine can read and write through dataX, a data synchronization tool

Small T Reading guide ：DataX It is an offline data synchronization tool open source by Alibaba cloud / platform , At present, it has supported the efficient data synchronization function including a variety of mainstream heterogeneous data sources . Our R & D team has completed  TDengine  The adaptation of , Related code has been DataX Officially accepted and incorporated into the trunk .

DataX  It is an offline data synchronization tool open source by Alibaba cloud / platform , At present, it has realized the following functions: MySQL、Oracle、OceanBase、SQL Server、PostgreSQL、HDFS、Hive、HBase And other heterogeneous data sources .

DataX The design idea is ：

DataX As a data synchronization framework , The synchronization of different data sources is abstracted as reading data from data sources  Reader  plug-in unit , And write data to the target  Writer  plug-in unit , Theoretically DataX The framework can support data synchronization of any data source type . meanwhile DataX Plug in system as a set of ecosystem , Each access to a new set of data sources , The interworking between the data source and the existing data source can be realized .

Based on this design idea , Our R & D team has completed TDengine The adaptation of , Realized  TDengineReader  and  TDengineWriter  Two plug-ins , And be DataX Official acceptance , Merged into its trunk .

TDengineReader  The features provided include ：

1. Supported by SQL Data filtering ;
2. Perform task segmentation according to time interval ;
3. Support TDengine All data types of ;
4. Support batch reading , Can pass  batchSize  Parameter controls the size of the batch pull result set , To improve read performance .

TDengineWriter  Supported features include ：

1. Support  OpenTSDB  Of json Line protocol in format , have access to TDengine Of Schemaless Write in TDengine.
2. Support batch write , Can pass  batchSize  Parameter controls the number of batch writes , This improves write performance .

at present TDengine Has joined DataX List of officially supported databases .

Now? , If the user wants to change the history Database（ such as MySQL、OpenTSDB etc. ） Data from to TDengine, Or will TDengine Data export in , You can use it DataX To implement the .

More usage details , You can refer to “ be based on DataX Of TDengine Data migration tools ” One article .

be based on DataX Of TDengineWriter and TDengineReader And the latest developments , Please pay attention to ：https://github.com/taosdata/DataX

 

---

Want to know more TDengine Database Specific details of , Welcome to GitHub View the relevant source code on .