当前位置:网站首页>The concept of spark independent cluster worker and executor
The concept of spark independent cluster worker and executor
2022-07-06 16:36:00 【Ruo Miaoshen】
List of articles
.
( One )Spark Independent clusters Worker and Executor The concept of
1.1 Worker( Containers )
Work node , Equivalent to workstation , A virtual computer , Have their own CPU The core number , Memory count .
We put Worker Suppose it is a computer , that CPU The core number ( hypothesis 2 nucleus ), Memory count ( hypothesis 32GB) Is its hardware condition .
In fact, Spark It is determined by these configurations :
export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=32g
1.2 Executor( Program )
Working procedure , Equivalent to a program running on a computer , It requires CPU The core number , Number of memory required .
We put Executor Suppose it is a program , Then it needs to run on the computer , It it's right CPU The core number ( hypothesis 1 nucleus ), Memory count ( hypothesis 4GB), Is the basic requirement of its operation .
PS: Empathy driver It's also a program .
We can specify these parameters when submitting :
SparkLauncher aL = new org.apache.spark.launcher.SparkLauncher()
.setXXX(...)
.setConf(SparkLauncher.DRIVER_MEMORY, "4g")
.setConf(SparkLauncher.EXECUTOR_MEMORY,"4g")
.setConf(SparkLauncher.EXECUTOR_CORES,"1")
1.3 Unable to meet the conditions Worker Don't work
If EXECUTOR_MEMORY
,EXECUTOR_CORES
The parameter specification exceeds some Worker Configuration of ,
Then these Work Will not participate in the calculation of this task .
If EXECUTOR_MEMORY
,EXECUTOR_CORES
The parameter specification exceeds all Worker Configuration of ,
Then there are no resources in the cluster for this task , The task will wait indefinitely , Wait for available resources .
Every worker You can configure multiple CPU The core , A certain amount of memory .
Configure according to the actual situation of the host , So here I am :
That's not a problem , It's just a concept .
.
边栏推荐
- SF smart logistics Campus Technology Challenge (no T4)
- Chapter 2 shell operation of hfds
- Codeforces Round #771 (Div. 2)
- Li Kou - 298th weekly match
- Raspberry pie 4B installation opencv3.4.0
- Problem - 1646C. Factorials and Powers of Two - Codeforces
- 图像处理一百题(1-10)
- js时间函数大全 详细的讲解 -----阿浩博客
- Market trend report, technological innovation and market forecast of double door and multi door refrigerators in China
- 去掉input聚焦时的边框
猜你喜欢
Problem - 922D、Robot Vacuum Cleaner - Codeforces
【锟斤拷】的故事:谈谈汉字编码和常用字符集
Raspberry pie 4b64 bit system installation miniconda (it took a few days to finally solve it)
Local visualization tools are connected to redis of Alibaba cloud CentOS server
Share an example of running dash application in raspberry pie.
简单尝试DeepFaceLab(DeepFake)的新AMP模型
Chapter 5 namenode and secondarynamenode
图像处理一百题(11-20)
提交Spark应用的若干问题记录(sparklauncher with cluster deploy mode)
VMware Tools和open-vm-tools的安装与使用:解决虚拟机不全屏和无法传输文件的问题
随机推荐
两个礼拜速成软考中级软件设计师经验
QT按钮点击切换QLineEdit焦点(含代码)
Date plus 1 day
(lightoj - 1349) Aladdin and the optimal invitation (greed)
input 只能输入数字,限定输入
(POJ - 3685) matrix (two sets and two parts)
OneForAll安装使用
QNetworkAccessManager实现ftp功能总结
Flag framework configures loguru logstore
Calculate the time difference
树莓派4B64位系统安装miniconda(折腾了几天终于解决)
Bisphenol based CE Resin Industry Research Report - market status analysis and development prospect forecast
Tert butyl hydroquinone (TBHQ) Industry Research Report - market status analysis and development prospect forecast
Installation and configuration of MariaDB
Codeforces Round #800 (Div. 2)AC
使用jq实现全选 反选 和全不选-冯浩的博客
Classic application of stack -- bracket matching problem
Research Report on market supply and demand and strategy of China's four seasons tent industry
Study notes of Tutu - process
力扣:第81场双周赛