当前位置:网站首页>Spark独立集群Worker和Executor的概念
Spark独立集群Worker和Executor的概念
2022-07-06 09:29:00 【若苗瞬】
.
(一)Spark独立集群Worker和Executor的概念
1.1 Worker(容器)
工作节点,相当于工作站,一台虚拟的计算机,有自己的CPU核心数,内存数。
我们把Worker假设成一台计算机,那么CPU核心数(假设2核),内存数(假设32GB)就是它的硬件条件。
实际上在Spark里面是这些配置决定的:
export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=32g
1.2 Executor(程序)
工作程序,相当于计算机上运行的一个程序,有它要求CPU核心数,要求的内存数。
我们把Executor假设成一个程序,那么它需要运行在计算机上,它它对CPU核心数(假设1核),内存数(假设4GB),就是它运行的基本要求。
PS:同理driver也是个程序。
我们提交时可以指定这些参数:
SparkLauncher aL = new org.apache.spark.launcher.SparkLauncher()
.setXXX(...)
.setConf(SparkLauncher.DRIVER_MEMORY, "4g")
.setConf(SparkLauncher.EXECUTOR_MEMORY,"4g")
.setConf(SparkLauncher.EXECUTOR_CORES,"1")
1.3 无法满足条件的Worker不工作
如果EXECUTOR_MEMORY
,EXECUTOR_CORES
参数指定超过了某些Worker的配置,
则这些Work不会参加到这个任务的计算中。
如果EXECUTOR_MEMORY
,EXECUTOR_CORES
参数指定超过了全部Worker的配置,
那么集群中就没有资源可以供这个任务使用了,任务会无限等待下去,等待可用资源。
每个worker可以配置多个CPU核心,一定量的内存。
根据主机实际情况进行配置,比如我这里:
这不是问题,只是个概念。
.
边栏推荐
- QT按钮点击切换QLineEdit焦点(含代码)
- 628. Maximum product of three numbers
- Pull branch failed, fatal: 'origin/xxx' is not a commit and a branch 'xxx' cannot be created from it
- Summary of game theory
- Flag framework configures loguru logstore
- Hbuilder X格式化快捷键设置
- 1605. Sum the feasible matrix for a given row and column
- Generate random password / verification code
- 409. Longest palindrome
- Codeforces Round #798 (Div. 2)A~D
猜你喜欢
1005. Maximized array sum after K negations
QT implementation window gradually disappears qpropertyanimation+ progress bar
MariaDB的安装与配置
软通乐学-js求字符串中字符串当中那个字符出现的次数多 -冯浩的博客
409. Longest palindrome
(POJ - 3685) matrix (two sets and two parts)
OneForAll安装使用
sublime text 代码格式化操作
Installation and configuration of MariaDB
Flask框架配置loguru日志库
随机推荐
Browser print margin, default / borderless, full 1 page A4
China double brightening film (dbef) market trend report, technical dynamic innovation and market forecast
(lightoj - 1323) billiard balls (thinking)
Codeforces Round #799 (Div. 4)A~H
1323. Maximum number of 6 and 9
Candy delivery (Mathematics)
Research Report on market supply and demand and strategy of Chinese table lamp industry
2078. Two houses with different colors and the farthest distance
QT实现圆角窗口
(POJ - 1458) common subsequence (longest common subsequence)
Acwing: Game 58 of the week
Codeforces round 797 (Div. 3) no f
Acwing - game 55 of the week
Opencv learning log 26 -- detect circular holes and mark them
Acwing: the 56th weekly match
(POJ - 3186) treatments for the cows (interval DP)
What is the difficulty of programming?
Hbuilder X格式化快捷键设置
Opencv learning log 29 -- gamma correction
Quick to typescript Guide