当前位置：网站首页>Dolphin-2.0.3 cluster deployment document
Dolphin-2.0.3 cluster deployment document
2022-06-12 21:56:00 【Interest e family】
DolphinScheudler-2.0.3 Cluster deployment documentation
1. Cluster service planning
Cluster machines ：hadoop1,hadoo2,hadoop3;
Cluster planning ：
2. Pre deployment environment preparation
Centos7.6 # The system environment of each machine is the same
Jdk1.8+ # Every machine has
MySQL-5.7.16 # hadoop3 On the machine
Zookeeper-3.5.7 # Cluster mode deployment , Every machine has
Hadoop-3.1.3 # Cluster mode deployment , Every machine has ,NameNode yes HA Pattern
Yarn-3.1.3 # HA Cluster mode deployment ,HA Corresponding machine ：hadoop1,hadoop2
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
3. Configure user password free login and permission
# stay hadoop1,hadoop2,hadoop3 The machine silly girl creates hadoop Users need to use root Sign in
useradd hadoop
# add to password
echo "hadoop" | passwd --stdin hadoop
# To configure sudo Unclassified
sed -i '$ahadoop ALL=(ALL) NOPASSWD: NOPASSWD: ALL' /etc/sudoers
sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
Be careful :
Because the task execution service is based on sudo -u {linux-user} Switching is different linux User's way to achieve multi tenant running jobs , So deployment users need to have sudo jurisdiction , And it's Secret free . If you find that /etc/sudoers In file "Defaults requirett" This line , Please comment out .
4. Machine configuration SSH Avoid secret landing
# stay hadoop1 The machine entered my home Catalog ,
su - hadoop
ssh-keygen -t rsa （ Press enter four times in succession ）
# After executing this order , Two files will be generated id_rsa（ Private key ）、id_rsa.pub（ Public key ）
# Copy the public key to the machine to log in without secret
ssh-copy-id hadoop1
ssh-copy-id hadoop2
ssh-copy-id hadoop3
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
Be careful : When the configuration is complete , You can run the command ssh hadoop1 Judge success , If you do not need to enter mima can ssh Landing proves successful .
5. start-up Zookeeper service
# stay hadoop1,hadoop2,hadoop3 Start up respectively
./bin/zkServer.sh start
- 1.
6. Download installation package
Installation package version ：2.0.3
# Download address ：
wget https://www.apache.org/dyn/closer.lua/dolphinscheduler/2.0.3/apache-dolphinscheduler-2.0.3-bin.tar.gz
- 1.
- 2.
take apache-dolphinscheduler-2.0.3-bin.tar.gz Put it in /home/hadoop/apps Under the table of contents
Use the previously created hadoop Users to perform subsequent operations .
7. decompression
# decompression tar package
tar -zxvf apache-dolphinscheduler-2.0.3-bin.tar.gz
- 1.
# rename
mv apache-dolphinscheduler-2.0.3-bin dolphinscheduler-2.0.3
- 1.
8. Modify the configuration file
vim dolphinscheduler-2.0.3/conf/config/install_config.conf
ips="hadoop1,hadoop2,hadoop3"
masters="hadoop1,hadoop2"
workers="hadoop1:default,hadoop2:default,hadoop3:default"
alertServer="hadoop1"
apiServers="hadoop2"
pythonGatewayServers="hadoop3"
installPath="/home/hadoop/data/dolphinschedulerInstallPath"
deployUser="hadoop"
dataBasedirPath="/home/hadoop/data/dolphinschedulerData"
javaHome="/opt/jdk1.8.0_212"
DATABASE_TYPE=${DATABASE_TYPE:-"mysql"}
SPRING_DATASOURCE_URL=${SPRING_DATASOURCE_URL:-"jdbc:mysql://hadoop3:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8"}
SPRING_DATASOURCE_USERNAME=${SPRING_DATASOURCE_USERNAME:-"dolphinscheduler"}
SPRING_DATASOURCE_PASSWORD=${SPRING_DATASOURCE_PASSWORD:-"dolphinscheduler"}
registryPluginName="zookeeper"
registryServers="hadoop1:2181,hadoop2:2181,hadoop3:2181"
registryNamespace="dolphinscheduler"
resourceUploadPath="/dolphinscheduler"
defaultFS="hdfs://ns1:8020"
resourceManagerHttpAddressPort="8088"
# yarn yes HA Mode with this
yarnHaIps="hadoop1,hadoop2"
singleYarnIp="yarnIp1"
# yarn No HA Mode with this
yarnHaIps=""
singleYarnIp="hadoop1"
hdfsRootUser="hadoop"
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
modify dolphinscheduler_env.sh, Configure according to the service on the machine .
vim dolphinscheduler-2.0.3/conf/env/dolphinscheduler_env.sh
export HADOOP_HOME=/home/hadoop/apps/hadoop-3.1.3
export HADOOP_CONF_DIR=/home/hadoop/apps/hadoop-3.1.3/etc/hadoop
export JAVA_HOME=/opt/jdk1.8.0_212
export HIVE_HOME=/home/hadoop/apps/hive-3.1.2
export PATH=$HADOOP_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$PATH
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
9. HDFS Profile copy
Hadoop Clustered NameNode Configured with HA, Need to open HDFS Type of resource upload , At the same time, we need to Hadoop Under the cluster core-site.xml and hdfs-site.xml Copied to the dolphinscheduler/conf, Not NameNode HA Skip the next step .
cp /home/hadoop/apps/hadoop-3.1.3/etc/hadoop/core-site.xml /home/hadoop/apps/dolphinscheduler-2.0.3/conf/
cp /home/hadoop/apps/hadoop-3.1.3/etc/hadoop/hdfs-site.xml /home/hadoop/apps/dolphinscheduler-2.0.3/conf/
- 1.
- 2.
10. New database 、 to grant authorization
DolphinScheduler Metadata is stored in a relational database , At present, we support PostgreSQL and MySQL, If you use MySQL You need to download it manually mysql-connector-java drive (8.0.16) And move to DolphinScheduler Of lib Under the table of contents . Let's say MySQL For example , Explains how to initialize the database
# Move mysql Of jar To lib Under the table of contents
cp mysql-connector-java-8.0.16.jar dolphinscheduler-2.0.3/lib
- 1.
# New database
mysql -uroot -p
mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
# to grant authorization
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'%' IDENTIFIED BY 'dolphinscheduler';
mysql>GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'localhost' IDENTIFIED BY 'dolphinscheduler';
mysql> flush privileges;
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
11. Copy directories to other machines
scp dolphinscheduler-2.0.3/ hadoop2:/home/hadoop/apps
scp dolphinscheduler-2.0.3/ hadoop3:/home/hadoop/apps
- 1.
- 2.
12. Initialize database
# stay hadoop1 Carry out the order
sh script/create-dolphinscheduler.sh
- 1.
13. start-up DolphinScheduler colony
stay hadoop1 Start on the machine DolphinScheduler colony
sh install.sh
- 1.
After startup jps see DolphinScheduler The related process
=====192.168.62.161=====
11607 MasterServer
11657 WorkerServer
11759 AlertServer
11697 LoggerServer
=====192.168.62.162=====
10585 MasterServer
10631 WorkerServer
11801 ApiApplicationServer
10671 LoggerServer
=====192.168.62.163=====
10410 WorkerServer
10275 LoggerServer
11850 PythonGatewayServer
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
14. Sign in DolphinScheduler
Browser access address http://hadoop2:12345/dolphinscheduler Log in to the system UI.
The default user name and mima yes admin/dolphinscheduler123
hadoop2 by ApiApplicationServer Location machine .
15. Start stop service
# One click to stop all services in the cluster
sh ./bin/stop-all.sh
# One click to start all services in the cluster
sh ./bin/start-all.sh
# Start stop Master
sh ./bin/dolphinscheduler-daemon.sh stop master-server
sh ./bin/dolphinscheduler-daemon.sh start master-server
# Start stop Worker
sh ./bin/dolphinscheduler-daemon.sh start worker-server
sh ./bin/dolphinscheduler-daemon.sh stop worker-server
# Start stop Api
sh ./bin/dolphinscheduler-daemon.sh start api-server
sh ./bin/dolphinscheduler-daemon.sh stop api-server
# Start stop Logger
sh ./bin/dolphinscheduler-daemon.sh start logger-server
sh ./bin/dolphinscheduler-daemon.sh stop logger-server
# Start stop Alert
sh ./bin/dolphinscheduler-daemon.sh start alert-server
sh ./bin/dolphinscheduler-daemon.sh stop alert-server
# Start stop Python Gateway
sh ./bin/dolphinscheduler-daemon.sh start python-gateway-server
sh ./bin/dolphinscheduler-daemon.sh stop python-gateway-server
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
边栏推荐
- 如何自己动手写一个vscode插件，实现插件自由！
- 图灵奖得主：想要在学术生涯中获得成功，需要注意哪些问题？
- Exception encountered by selenium operation element
- SQL tuning guide notes 13:gathering optimizer statistics
- SQL调优指南笔记11：Histograms
- MySQL architecture and basic management (II)
- The ifeq, filter and strip of makefile are easy to use
- [qnx hypervisor 2.2 manuel de l'utilisateur] 4.2 environnement de construction pris en charge
- jsonUtils
- Ansible summary (VI)
猜你喜欢
Oracle livelabs experiment: introduction to Oracle Spatial
建立高可用的数据库
The ifeq, filter and strip of makefile are easy to use
SQL tuning guide notes 9:joins
About the solution to "the application cannot start normally 0xc00000022" after qt5.15.2 is installed and qtcreator is started
Compiling process of OpenSSL and libevent on PC
Design and practice of Hudi bucket index in byte skipping
How to prevent phishing emails? S/mime certificate to help!
"Oracle database parallel execution" technical white paper reading notes
KDD2022 | GraphMAE:自监督掩码图自编码器
随机推荐
Oracle LiveLabs实验：Introduction to Oracle Spatial Studio
建立高可用的数据库
图灵奖得主：想要在学术生涯中获得成功，需要注意哪些问题？
SQL tuning guide notes 10:optimizer statistics concepts
[simple] 155 Minimum stack
Ansible summary (VI)
复杂系统如何检测异常？北卡UNCC等最新《复杂分布式系统中基于图的深度学习异常检测方法综述》，阐述最新图异常检测技术进展
[Jianzhi offer simple] Jianzhi offer 06 Print linked list from end to end
动态规划之如何将问题抽象转化为0-1背包问题（详解利用动态规划求方案数）
Data batch writing
[sword finger offer] sword finger offer 35 Replication of complex linked list
KDD2022 | GraphMAE:自监督掩码图自编码器
Oracle数据库中查询执行计划的权限
SQL调优指南笔记10：Optimizer Statistics Concepts
OceanBase 社区版 OCP 功能解读
Gzip compression decompression
June training (day 11) - matrix
CVPR 2022 | 应对噪声标签，西安大略大学、字节跳动等提出对比正则化方法
SQL tuning guide notes 9:joins
Logstash timestamp converted to UNIX nanosecond nano second time