当前位置:网站首页>[yarn] CDP cluster yarn configuration capacity scheduler batch allocation
[yarn] CDP cluster yarn configuration capacity scheduler batch allocation
2022-07-06 11:32:00 【kiraraLou】
One 、 Preface
It's going to be upgraded recently CDH
Cluster into CDP
colony ,CDH
In the cluster Yarn
By default, the service uses fair
Scheduler ,CDP
The cluster uses capacity
Scheduler , We've been there before The scheduler is set unreasonably due to batch allocation , As a result, tasks are allocated to certain nodes , Make the cluster resource load extremely unbalanced .
To avoid the same problem in CDP
On the cluster , We conduct research in advance . Look at using Will the scheduler also have centralized allocation . But in the process of research , There are some unexpected phenomena , Waiting for follow-up .
Two 、CDH Centralized distribution
As mentioned earlier ,CDH 5.8-Hadoop2.6.0
After version , in the light of Fair dispatch
, There are several configurations as follows , For task types ( Small tasks ) Accelerate the optimization of allocation .
Configuration name | explain |
---|---|
yarn.scheduler.fair.max.assign | Maximum allocation : If assignmultiple by true And dynamic.max.assign by false, Then the maximum number of containers that can be allocated in a heartbeat . |
yarn.scheduler.fair.assignmultiple | Assign multiple : Whether multiple containers are allowed to be allocated in a heartbeat . |
yarn.scheduler.fair.dynamic.max.assign | If assignmultiple It's true , Whether to dynamically determine the amount of resources that a heartbeat can allocate . After opening , About half of the unallocated resources on the node will be allocated to the container in a heartbeat . Default to true . |
Through reasonable configuration , We can use centralized allocation , It will not expand the cluster load difference .
CDH
How to configure a cluster is not covered here .
3、 ... and 、CDP Centralized distribution
CDP Already used in the cluster Capacity scheduling
As the default scheduler , By consulting the official and Cloudera file , Find out Scheduling is also possible through heartbeat NodeManager
Allocate multiple containers . The configuration is as follows :
Configuration name | explain |
---|---|
yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enabled | Whether to allow in a NodeManager Allocate multiple containers in the heartbeat . Default to true . |
yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments | If multiple-assignments-enabled by true , In a NodeManager The maximum number of containers that can be allocated in the heartbeat . The default is -1, No restrictions . |
yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments | If multiple-assignments-enabled by true , In a NodeManager The maximum that can be allocated in the heartbeat off-switch Number of containers . The default is 1, Indicates that only one off switch is allowed to be assigned in a heartbeat . |
How to configure
stay
Cloudera Manager
in , Select cluster >YARN Queue manager UI service
.stay
YARN
In the queue manager window , Click the scheduler configuration tab .stay “ Scheduler configuration ” Window
Choose
Enable Multiple Assignments Per Heartbeat
Check box to allow in aNodeManager
Allocate multiple containers in the heartbeatConfigure the following
NodeManager
Heartbeat properties :
Maximum Container Assignments Per Heartbeat
: In aNodeManager
The maximum number of containers that can be allocated in the heartbeat . Set this value to -1 This restriction will be disabled .Maximum Off-Switch Assignments Per Heartbeat
: Can be in aNodeManager
The maximum number of closed switch containers allocated in the heartbeat .
3、 ... and 、 summary
capacity
Schedulers have similarfair
The heartbeat batch allocation configuration of the scheduler .- CDP colony
capacity
The scheduler enables batch allocation by default , And the assigned quantity is 100, This value needs to be reduced . - Now it's tested , It is found that the configuration does not seem to be effective , This needs to be followed up by experts .
Reference resources
https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/yarn-allocate-resources/topics/yarn-set-user-limits.html
https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/yarn-allocate-resources/topics/yarn-configure-nm-heartbeat.html
https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Reviewing_the_configuration_of_the_CapacityScheduler
边栏推荐
- [Blue Bridge Cup 2017 preliminary] grid division
- When you open the browser, you will also open mango TV, Tiktok and other websites outside the home page
- L2-006 tree traversal (25 points)
- vs2019 使用向导生成一个MFC应用程序
- QT creator test
- ES6 promise object
- When using lambda to pass parameters in a loop, the parameters are always the same value
- Software I2C based on Hal Library
- Library function -- (continuous update)
- Codeforces Round #753 (Div. 3)
猜你喜欢
{一周总结}带你走进js知识的海洋
分布式节点免密登录
Error connecting to MySQL database: 2059 - authentication plugin 'caching_ sha2_ The solution of 'password'
解决安装Failed building wheel for pillow
Solve the problem of installing failed building wheel for pilot
Rhcsa certification exam exercise (configured on the first host)
[download app for free]ineukernel OCR image data recognition and acquisition principle and product application
MTCNN人脸检测
Valentine's Day flirting with girls to force a small way, one can learn
Face recognition_ recognition
随机推荐
数据库高级学习笔记--SQL语句
MySQL and C language connection (vs2019 version)
QT creator support platform
【CDH】CDH5.16 配置 yarn 任务集中分配设置不生效问题
[蓝桥杯2017初赛]包子凑数
Reading BMP file with C language
Aborted connection 1055898 to db:
误删Path变量解决
Attention apply personal understanding to images
[number theory] divisor
error C4996: ‘strcpy‘: This function or variable may be unsafe. Consider using strcpy_ s instead
{一周总结}带你走进js知识的海洋
ES6 let 和 const 命令
SQL时间注入
【yarn】Yarn container 日志清理
Vs2019 use wizard to generate an MFC Application
ImportError: libmysqlclient. so. 20: Cannot open shared object file: no such file or directory solution
Niuke novice monthly race 40
Codeforces Round #771 (Div. 2)
Vs2019 first MFC Application