How to deploy dolphin scheduler Apache dolphin scheduler 1.2.0 in cdh5.16.2

Apache Dolphin Scheduler

Component is introduced

Distributed and extensible visualization DAG Workflow task scheduling system . Committed to solving the complex dependencies in the data processing process , Make scheduling system use out of the box in data processing flow .

Official website : https://dolphinscheduler.apache.org/en-us/

Github : https://github.com/apache/incubator-dolphinscheduler

Deployment environment

• CDH Test environment
  • 6 Taiwan machine
  • Gateway node deployment worker
  • CM Node deployment master And monitoring web
  • The gateway node has been deployed hive&spark gateway
• Platform version
  • CDH5.16.2
  • Dolphin Scheduler 1.2.0
• Basic software
  • PostgreSQL or MySql Store metadata

Front end deployment

Installation package download

https://dolphinscheduler.apache.org/en-us/docs/release/download.html

• Create deployment folder /opt/ds, Upload tar Package into this directory , And unpack it

# create deploy dir
mkdir -p /opt/ds/ds-ui;
# decompression
tar -zxvf apache-dolphinscheduler-incubating-1.2.1-SNAPSHOT-dolphinscheduler-front-bin.tar.gz -C /opt/ds/;
mv apache-dolphinscheduler-incubating-1.2.1-SNAPSHOT-dolphinscheduler-front-bin ds-1.2.0-ui;

Choose Automated Deployment

• Check yum Source , Here is the development environment , The external network needs to use a proxy , Need to install nginx
• Get into ds-1.2.0-ui Catalog , perform install-dolphinscheduler-ui.sh set up script
• Modify the front-end port to ：8886, Prevent and Hue Port conflict
• modify api-server Of ip
• modify api-server port
• choice centos7 install

modify nginx Upload size parameters

• add to nginx To configure client_max_body_size 1024m;
• restart nginx
• This step must be done , Otherwise, the resource is too large to upload to the resource center

vi /etc/nginx/nginx.conf

# add param
client_max_body_size 1024m;

# restart nginx
systemctl restart nginx

Visit the front page 8888（ Custom becomes 8886） port , appear loading page , front end web installation is complete

Back end deployment

preparation

Download installation package

https://dolphinscheduler.apache.org/en-us/docs/release/download.html

Upload tar Package to /opt/ds in , And unpack it

tar -zxvf apache-dolphinscheduler-incubating-1.2.1-SNAPSHOT-dolphinscheduler-backend-bin.tar.gz -C /opt/ds/;
mv apache-dolphinscheduler-incubating-1.2.1-SNAPSHOT-dolphinscheduler-backend-bin ds-1.2.0-backend;

Create deployment user

• Create the deployment user and set the password ( All deployment machines )
• Hang the deployment user to hadoop Under the group , Use hdfs As a Resource Center
• To configure sudo Unclassified

# add user dscheduler
useradd dscheduler;
# modify user password
passwd dscheduler;
# add sudo
vi /etc/sudoers;
dscheduler      ALL=(ALL)       NOPASSWD: ALL

• Switch to the deployment user and configure the machine to log in without secret , The pseudo distributed system needs to be configured with local password free login

su dscheduler;
ssh-keygen -t rsa;
# Configure mutual security free and stand-alone security free ,[hostname] Configure the machine that needs to be secret free hosts
ssh-copy-id -i ~/.ssh/id_rsa.pub [email protected][hostname];

Database initialization

• Get into CDH Clustered mysql
  • mysql -uroot -p
• The default database is pg,mysql Need to add mysql-connector-java Package to lib Under the table of contents
• Execute database initialization command , Set the access account password

CREATE DATABASE dscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON dscheduler.* TO 'dscheduler'@'%' IDENTIFIED BY 'xxxx';
GRANT ALL PRIVILEGES ON dscheduler.* TO 'dscheduler'@'localhost' IDENTIFIED BY 'xxxx';
FLUSH PRIVILEGES;

• Create tables and import basic data
  • modify conf In the catalog application-dao.properties file
  • Comment out pg, Use mysql
  • add to mysql-connector-java Package to lib Under the table of contents

-  perform script In the catalog create-dolphinscheduler.sh

Configure environment variables

• Modify directory permissions

chown -R dscheduler:dscheduler ds-1.2.0-backend/;
chmod -R 755 ds-1.2.0-backend/;

• modify conf/env In the catalog .dolphinscheduler_env.sh file
  • ds-1.2.0 In version Spark Task component , You can only submit Spark1 The task of
  • SPARK_HOME1&SPARK_HOME2 All are configured as cluster Spark2-Home
  • You can also comment out SPARK_HOME1
  • Flink Not deployed in the cluster , Do not modify parameters

• take jdk Soft chain to /bin/java Next

ln -s /usr/java/jdk1.8.0_131/bin/java /usr/bin/java

• modify install.sh Configuration of , According to the cluster itself
  • Pay attention to the parameters
    • installPath - take ds Where to install it , Such as ：/opt/ds-agent
    • zkQuorum - It must be ip:2181, Remember to 2181 Port with
    • deployUser - Deploy users , Action required HDFS Authority
  • To use HDFS As a Resource Center ,HA Under the circumstances , Need to cluster core-site.xml Document and hdfs-site.xml File copy to conf Catalog

Deployment installation kazoo

• install python Of zk Tools
  • CDH Cluster defaults to python2.7

yum -y install python-pip;
pip install kazoo;

• perform install Script ,sh install.sh
• stay worker and master Use on the machine jps Check whether the service is started

• Access the front end
  • user name admin
  • password dolphinscheduler123

• dolphin scheduler 1.2.0 Deployment completed

DAG test

• Create tenants

• Create user
• If there is a problem with tenant creation , Please check whether content center is enabled

• Create a new project and a new workflow

• Run the workflow , View the execution results

• thus Dolphin Scheduler 1.2.0 dag demo Testing is completed