The concept of spark independent cluster worker and executor

.

（ One ）Spark Independent clusters Worker and Executor The concept of

1.1 Worker（ Containers ）

Work node , Equivalent to workstation , A virtual computer , Have their own CPU The core number , Memory count .
We put Worker Suppose it is a computer , that CPU The core number （ hypothesis 2 nucleus ）, Memory count （ hypothesis 32GB） Is its hardware condition .
In fact, Spark It is determined by these configurations ：

export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=32g

1.2 Executor（ Program ）

Working procedure , Equivalent to a program running on a computer , It requires CPU The core number , Number of memory required .
We put Executor Suppose it is a program , Then it needs to run on the computer , It it's right CPU The core number （ hypothesis 1 nucleus ）, Memory count （ hypothesis 4GB）, Is the basic requirement of its operation .

PS: Empathy driver It's also a program .

We can specify these parameters when submitting ：

SparkLauncher aL = new org.apache.spark.launcher.SparkLauncher()
				.setXXX(...)
                .setConf(SparkLauncher.DRIVER_MEMORY, "4g")
                .setConf(SparkLauncher.EXECUTOR_MEMORY,"4g")
                .setConf(SparkLauncher.EXECUTOR_CORES,"1")

1.3 Unable to meet the conditions Worker Don't work

If EXECUTOR_MEMORY,EXECUTOR_CORES The parameter specification exceeds some Worker Configuration of ,
Then these Work Will not participate in the calculation of this task .

If EXECUTOR_MEMORY,EXECUTOR_CORES The parameter specification exceeds all Worker Configuration of ,
Then there are no resources in the cluster for this task , The task will wait indefinitely , Wait for available resources .

Every worker You can configure multiple CPU The core , A certain amount of memory .
Configure according to the actual situation of the host , So here I am ：

---

That's not a problem , It's just a concept .

.