08 Spark cluster construction

Foreword

Ha ha recently there are a series of related requirements for environment construction

Record it

spark three nodes: 192.168.110.150, 192.168.110.151, 192.168.110.152

150 is master, 151 is slave01, 152 is slave02

All three machines are trusted shell

spark version is spark-3.2.1-bin-hadoop2.7

Spark cluster setup

spark three nodes: 192.168.110.150, 192.168.110.151, 192.168.110.152

1. Basic environment preparation

Install jdk on 192.168.110.150, 192.168.110.151, 192.168.110.152, and upload the spark installation package

The installation package is from Downloads | Apache Spark

2. spark configuration adjustment

Copy the following three configuration files, make adjustments, and then scp to slave01, slave02 above

[email protected]:/usr/local/ProgramFiles/spark-3.2.1-bin-hadoop2.7# cp conf/spark-defaults.conf.template conf/[email protected]:/usr/local/ProgramFiles/spark-3.2.1-bin-hadoop2.7# cp conf/spark-env.sh.template conf/[email protected]:/usr/local/ProgramFiles/spark-3.2.1-bin-hadoop2.7# cp conf/workers.template conf/workers

Update workers

# A Spark Worker will be started on each of the machines listed below.slave01slave02

Update spark-defaults.conf

spark.master spark://master:7077# spark.eventLog.enabled true# spark.eventLog.dir hdfs://namenode:8021/directoryspark.serializer org.apache.spark.serializer.KryoSerializerspark.driver.memory 1g# spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"

Update spark-env.sh

export JAVA_HOME=/usr/local/ProgramFiles/jdk1.8.0_291export HADOOP_HOME=/usr/local/ProgramFiles/hadoop-2.10.1export HADOOP_CONF_DIR=/usr/local/ProgramFiles/hadoop-2.10.1/etc/hadoopexport SPARK_DIST_CLASSPATH=$(/usr/local/ProgramFiles/hadoop-2.10.1/bin/hadoop classpath)export SPARK_MASTER_HOST=masterexport SPARK_MASTER_PORT=7077

3. Start the cluster

The machine where the master is located executes start-all.sh

[email protected]:/usr/local/ProgramFiles/spark-3.2.1-bin-hadoop2.7# ./sbin/start-all.shstarting org.apache.spark.deploy.master.Master, logging to /usr/local/ProgramFiles/spark-3.2.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.outslave01: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/ProgramFiles/spark-3.2.1-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave01.outslave02: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/ProgramFiles/spark-3.2.1-bin-hadoop2.7/logs/spa[email protected]master:/usr/local/ProgramFiles/spark-3.2.1-bin-hadoop2.7#

Test cluster

Submit 1000 iterations of SparkPI using spark-submit

spark-submit --class org.apache.spark.examples.SparkPi /usr/local/ProgramFiles/spark-3.2.1-bin-hadoop2.7/examples/jars/spark-examples_2.12-3.2.1.jar 1000

java driver submits spark task

spark web ui monitoring page

End