Spark cluster deployment (third bullet)

目录

                               spark集群部署

1 修改spark-env.sh配置文件

2 启动spark HA集群

3.测试spark HA集群,To demonstrate whether to solve the single point of failure problem,关闭master节点的master进程

---

       

                                    spark集群部署

1 修改spark-env.sh配置文件

$ssh master

$cd opt/module/spark/conf

$vi spark-env.sh

步骤1 将指定masterNode configuration parameter comments‘#’,添加SPARK_DAEMON_JAVA_OPTS配置参数

export JAVA_HOME=/opt/module/jdk1.8.0_144

#export SPARK_MASTER_HOST=master

export SPARK_MASTER_PORT=7077

export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER

-Dspark.deploy.zookeeper.url=master:2181,slave1:2181,slave2:2181

-Dspark.deploy.zookeeper.dir=/spark"

（1）spark.deploy.recoveryMode=ZOOKEEPER：设置zookeeperto enable backupmaster模式

（2）spark.deploy.zookeeper.url=master：指定zookeeper的server地址

（3）spark.deploy.zookeeper.dir：The files and directories that hold cluster metadata information

将spark-env.sh分发至slave1和slave2节点上,Ensure that the configuration files are unified

scp spark-env.sh slave1:/opt/module/spark/conf/

scp spark-env.sh slave2:/opt/module/spark/conf/

 

2 启动spark HA集群

步骤1 启动zookeeper服务,3台Nodes are required

$cd /opt/module/zookeeper/bin

$ zkServer.sh start

问题1 : 启动sparkThe cluster has the following problem

解决：配置问题,自己把spark-env.shThe horizontal bar in the configuration content is empty

问题2  在slave1节点启动master服务报 stop it first（It means you need to stop this first,才能开启master,You can also not stop）

解决：在master节点上停止spark集群

$cd opt/software/spark

$sbin/stop-all.sh

After solving the above problems,重新在master节点启动sparkcluster andslave节点单独启动master服务

在masterLaunched separately on the nodespark服务

$/opt/module/spark/sbin/start-all.sh

在slave1Launched separately on the nodeMaster服务

$/opt/module/spark/sbin/start-master.sh

步骤2 访问http://192.168.196.102（slave1）:8080,查看备用master节点状态

Status：STANDBY  # 说明spark HA配置完成

3.测试spark HA集群,To demonstrate whether to solve the single point of failure problem,关闭master节点的master进程

$ /opt/module/spark/sbin/stop-master.sh