ETL tool (data synchronization) II

kettle After using and mastering , Also need to master linux The strategy of regular execution under

One 、kettle It feels convenient to use

although kettle Bring your own scheduling task . But the premise is to keep kettle open . And the job is running . Not suitable for remote server installation windows client , Only for learning

stay linux install kettle And scheduling is a better method

Two 、 Use steps

1、 take kettle Folder data-integration  Unzip to linux Server establish /kettle/ Directory storage program

2、 Establish directory storage kettle file

mkdir /data/kettle/kettle_job // Store operation documents 
mkdir /data/kettle/kettle_transition // Storage conversion 
mkdir /data/kettle/kettle_sh // Store the execution script 
mkdir /data/kettle/kettle_log // Storage execution kettle Generate log 

Will be taken from windows Configure the generated .ktr and .kjb The programs are placed in transition Catalog and job Catalog

3、 Write execution scripts

/data/kettle/kettle_sh/start.sh
 

#!/bin/bash

# perform job Program 

cd /data/kettle/data-integration

export JAVA_HOME=/usr/local/jdk1.8.0_201/

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$PATH:$JAVA_HOME/bin

./kitchen.sh -file=/data/kettle/kettle_job/testjob.kjb >>/data/kettle/kettle_log/ceshi_$(date +%Y%m%d).log

notes ：( because crond It's a daemon , It does not belong to any user , Although before root The user has configured java environment variable , however crond I can't find java command , therefore , When crond perform kettle When the task , Need dynamic settings java environment variable , Otherwise, manual execution will occur .sh That's all right. , discharge crontb There is no effect after execution )

preservation start.sh, Execute the command granting permission

chmod +x start.sh

Test whether the script can execute normally

sh -x start.sh

4、 Set up crontab Scheduled tasks

# crontab -e

*/1 * * * * /data/kettle/kettle_sh/ceshi.sh // Execute every minute ceshi.sh

# systemctl restart crond

# systemctl enable crond

Effective completion

3、 ... and 、ETL kettle Learning path

1、ETL Tools ( Data synchronization )_ Invincible OTA's blog -CSDN Blog Kettle Do cross database data migration https://blog.csdn.net/qq_36602951/article/details/125898662

2、

ETL Tools ( Data synchronization ) Two _ Invincible OTA's blog -CSDN Blog kettle After using and mastering , Also need to master linux The strategy of regular execution under https://blog.csdn.net/qq_36602951/article/details/125959374