DataX User Guide

DataX

DataX It is an offline data synchronization tool widely used in Alibaba group / platform , Implementation include MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、DRDS And other efficient data synchronization functions among various heterogeneous data sources .
DataXgit Address

install

Direct download DataX tool kit ：DataX Download address
Download and unzip to a local directory , Get into bin Catalog , You can run the synchronization job ：
shell cd {YOUR_DATAX_HOME}/bin python datax.py {YOUR_JOB.json}
Self test script ：
python {YOUR_DATAX_HOME}/bin/datax.py {YOUR_DATAX_HOME}/job/job.json

Use

1. Generate job.json Example file for

cd  {
    YOUR_DATAX_HOME}/bin
python datax.py -r mysqlreader -w postgresqlwriter

The instance file will be generated

2. Copy the instance file to .json In the document . Modify the configuration items , example json The file is shown below ：

{
  "job": {
    "content": [
      {
        "reader": {
          "name": "mysqlreader",
          "parameter": {
            "column": ["id" , "fid" , "name" , "type" , "ab"],
            "connection": [
              {
                "jdbcUrl": ["jdbc:mysql://192.168.0.111:3306/etl"],
                "table": ["district"]
              }
            ],
            "password": "root",
            "username": "root",
            "where": ""
          }
        },
        "writer": {
          "name": "postgresqlwriter",
          "parameter": {
            "column": ["id" , "fid" , "name" , "type" , "ab"],
            "connection": [
              {
                "jdbcUrl": "jdbc:postgresql://192.168.0.111:5432/etl",
                "table": ["district"]
              }
            ],
            "password": "root",
            "postSql": [],
            "preSql": [],
            "username": "postgres"
          }
        }
      }
    ],
    "setting": {
      "speed": {
        "channel": 5
      }
    }
  }
}

instructions

1. DataX Unable to synchronize table structure , So when synchronizing data , Must ensure writer Table in already exists .
2. column It's best not to use... For fields “*” Instead of , Make sure that you can write whatever fields you need to synchronize .WARN: There is some risk in the column configuration in your configuration file . Because you are not configured to read the columns of the database table , When the number of fields in your table 、 When the type changes , It may affect the correctness of the task and even run in error .
3. Writing to the database is generally not recommended , Number of channels > 32