Deploy dolphin scheduler high availability cluster based on rainbow

In this paper, we describe the process by Rainbond Cloud native application management platform One click deployment highly available DolphinScheduler colony , This way is suitable for people who don't know much about Kubernetes、 Users of complex technologies such as containerization , Reduced in Kubernetes Deployment in China DolphinScheduler Threshold .

Apache DolphinScheduler It's a distributed and extensible visualization DAG Workflow task scheduling open source system . Solve data research and development ETL Intricate dependencies , Can't directly monitor task health status and other issues .DolphinScheduler With DAG The streaming approach will Task Assemble , It can monitor the running status of tasks in real time , At the same time, it supports retry 、 Recovery from the specified node failed 、 Suspension and Kill Tasks, etc

Simple and easy to use ：DAG Monitoring interface , All process definitions are visual , Customize by dragging tasks DAG, adopt API Mode docking with the third party system , One key deployment

high reliability ： A lot of decentralization Master And many Worker, Self support HA function , Use task queue to avoid overload , It will not cause the machine to get stuck

Rich use scenarios ： Support pause resume operation . Multi tenant support , Better response to big data usage scenarios . Support more task types , Such as spark, hive, mr, python, sub_process, shell

High scalability ： Support custom task type , Scheduler uses distributed scheduling , Scheduling capacity grows linearly with cluster ,Master and Worker Support dynamic online and offline

Prerequisite

• Usable Rainbond Cloud native application management platform , See documentation Rainbond Fast installation

DolphinScheduler Cluster one click deployment

• Dock and visit the built-in open source app store , Search keywords dolp You can find it DolphinScheduler application .

• Click on DolphinScheduler On the right side of the install Go to the installation page , Fill in the corresponding information , Click OK to start the installation , Automatically jump to the application view .

Options	explain
Team name	User built workspace , Isolate by namespace
Cluster name	choice DolphinScheduler To which one K8s colony
Select application	choice DolphinScheduler To which application is deployed , The application contains several related components
Application version	choice DolphinScheduler Version of , Currently, the optional version is 3.0.0-beta2

• Wait a few minutes ,DolphinScheduler The cluster will be installed , And run it .

• Click on the access , Will visit DolphinScheduler-API Components , The default user password is admin / dolphinscheduler123

API Master Worker Node scaling

DolphinScheduler API、Master、Worker Both support scaling multiple instances , Multiple instances can ensure the high availability of the entire cluster .

With Worker For example , Enter the assembly -> Telescopic , Set the number of instances .

verification Worker node , Get into DolphinScheduler UI -> The monitoring center -> Worker View node information .

The configuration file

API and Worker Service sharing /opt/dolphinscheduler/conf/common.properties , When modifying the configuration, you only need to modify API The configuration file for the service .

How to support Python 3？

Worker The service is installed by default Python3, You can add environment variables when using PYTHON_HOME=/usr/bin/python3

How to support Hadoop, Spark, DataX etc. ？

With Datax For example ：

1. Installing a plug-in .Rainbond Team view -> plug-in unit -> Install plug-ins from the store -> Search for Universal data initialization plug-in And install .
2. Open plug-ins . Get into Worker In component -> plug-in unit -> Opening Universal data initialization plug-in , And modify the configuration
   • FILE_URL：http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz
   • FILE_PATH：/opt/soft
   • LOCK_PATH：/opt/soft
3. Update components , The initialization plug-in will be downloaded automatically Datax And unpack it /opt/soft Under the table of contents .