Multi graph explanation of resource preemption in yarn capacity scheduling

In the previous article, we talked about the capacity configuration of queues in capacity scheduling 、 Priority scheduling in capacity scheduling .

There may be several situations in the actual use scenario ：

• The resources used by the running tasks in a queue exceed the configured capacity of the queue , Then tasks submitted to other queues may not run because of insufficient resources （ You can only wait until the end of the running task to release resources ）

• Again , Multiple running low priority tasks occupy a lot of cluster resources , As a result, the newly submitted high priority task cannot run .

These scenarios can be solved by configuring resource preemption . Ensure that the minimum capacity of the queue is guaranteed 、 Or high priority tasks run first .

This article will talk about resource preemption in capacity scheduling .

【 Preemption between queues 】

---

Resource preemption in capacity scheduling , The most common way is to preempt resources among multiple queues , Ensure the minimum resources for each queue （ Queued capacity To configure ） To be satisfied .

stay rm Configuration file for yarn-site.xml The following configuration is added in , You can start the preemption between queues ：

<property> <name>yarn.resourcemanager.scheduler.monitor.enable</name>    <value>true</value></property>

When the configuration is complete （ Need to restart rm）, You can see the preemption attribute of the queue from disable become enabled 了 .

Next , We will actually test the effect of preemption .

The queue configuration of the test is as follows ：

	queue_test	spurs	default
capacity	10	40	50
maxCapacity	100	100	100

notes ： The total resources of the cluster are 18GB Memory , In order to submit and allocate resources correctly , Temporarily queue am The proportion of resources has been increased .

testing procedure ：

• Use three users to queue_test Queue commit 3 A mission , That is, each user submits a task （ In order to ensure that each task can be allocated to resource operation ）, The task consists of two container, Total applications 5GB Memory （3 Tasks occupy 15GB resources ）.

• After the tasks have been successfully allocated resources to run , And then to other queues （ for example default queue ） Submit a task （ Total applications 5GB Memory ）, Observe whether preemption will be triggered .

The test is as follows ：

queue_test There are already... In the queue 3 A mission , And they have successfully allocated resources and run ：

Submit a new task to default After queue ,am Resources have been allocated and successfully run , But at this time, due to insufficient cluster resources , As a result, the task cannot apply to the resource .

Wait a while , It is observed that the newly submitted task starts from queue_test The queue preempted 3GB Resources for , and queue_test One of the tasks in the queue finished running and released resources .

from rm The information provided by the interface of , Preemption did happen .

Here we are , You can see that the preemption has taken effect . After that, if you continue to default Queue or spurs Queue submit task , Will continue from queue_test Preempt resources in .

【 Preemption in the queue 】

---

In addition to configuring preemption between queues , You can also configure whether tasks in the queue can preempt . In other words, it can not only preempt the resources of other queues , You can also preempt the resources of the task queue .

There are two ways to preempt in the queue ： One is based on the priority of the task , That is, after the high priority task is in the status to be allocated , Will preempt resources from low priority tasks ; The other is divided by user resources , Tasks submitted by multiple users in the queue , Preempt from the user who occupies the most resources , Its essence is to ensure that each user's resource use tends to be equal . Here is an example of preemption by task priority .

It is similar to preemption between queues , It also needs to be yarn-site.xml The following configuration items can be added to enable preemption in the queue .

<property> <name>yarn.resourcemanager.monitor.capacity.preemption.infra-queue-preemption.enabled</name> <value>true</value></property>
<property>    <name>yarn.resourcemanager.monitor.capacity.preemption.intra-queue-preemption.preemption-order-policy</name>    <value>priority_first</value></property>

When the configuration is complete （ Also restart rm）, You can see the corresponding attributes from disable become enabled 了 .

The next step is still to test and see the actual effect , The test method is basically consistent with the preemption between queues , But there are two differences ：

• The last task submission is still submitted to queue_test In line （ Preemption between queues is committed to default queue ）

• Each task is submitted with a priority , The last task submitted has a higher priority than the previous one

The test is as follows ：

queue_test There are already... In the queue 3 A mission , And they have successfully allocated resources and run , Submit the task to the queue again , The priority of this task is higher than that of the previous , Mission am Resources already allocated , However, because the cluster resources are not satisfied, resources will not be allocated to tasks （ The screenshot here is a little early , The task will run after allocating resources , Its state will change to RUNNING）

After a while , An inter queue preemption has occurred , One of the lowest priority tasks container Ended and released resources .

thereafter , Submit a task to the queue again （ The priority is lower than that of running ）, I found that there was no preemption for a long time . This indicates that only high priority tasks can preempt the resources of low priority tasks , But not vice versa .

【 Preemption principle 】

---

First , Only the scheduler used implements PreemptableResourceScheduler Interface , And preemption is enabled ;rm Will really use the preemption function .

Resource preemption is triggered by a third-party policy , These strategies are usually implemented as pluggable component classes （ Implement the specified SchedulingEditPolicy Interface ）.yarn The default implementation class is provided , Of course , It can also be specified through parameter configuration .

rm Will start a monitoring thread , These policies are traversed periodically in this thread , And call the interface implementation method of the specific instance , Decide whether to preempt , Seize what container Resources for .

The whole process of resource preemption can be summarized as the following steps ：

1. The monitoring thread is based on the current used resource size of the queue 、 The actual configuration uses the resource size 、 Whether to allow preemption and other factors , Recalculate the final allocated resource size for each queue , The size of resources to be preempted , And what container Will be preempted . Then, the resources that need to be preempted are notified through the event mechanism rm.

2. rm Handle the corresponding event , And mark those to be preempted container.

3. rm Mission received ApplicationMaster（ abbreviation AM） After the heartbeat information of , The total amount of resources to be released and resources to be preempted will be through heartbeat response container The list is returned to AM.AM After receiving the heartbeat , The following operation modes can be selected ：

   a. Kill these container

   b.  Kill others container To make up the total amount of resources to be released

   c.  Do nothing , Because there may be other container End self releasing resources , Or by rm Choose to kill container

4. After the timer corresponding to the monitoring thread expires , Find out AM Not according to the specified list kill To be seized container, Will be sent containing these container List kill Event to rm.

5. rm Mark these after receiving the message container Will be kill. And in NodeManager In the heartbeat response , inform NodeManager You need to put the specified container Conduct kill, To release resources .

This is the processing logic of the whole resource preemption , Personally think that , The core step is Recalculation and annotation of resource allocation container Our resources will be preempted .

【 Other configuration items 】

---

In addition to the above queue preemption 、 Queue preemption is beyond the configuration mentioned , There are also the following related configuration items ：

<!--  Set to observation mode ,true It means no real preemption  --><property> <name>yarn.resourcemanager.monitor.capacity.preemption.observe_only</name> <value>true</value></property>
<!--  Resource preemption strategy  --><property> <name>yarn.resourcemanager.scheduler.monitor.policies</name>    <value>org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy</value></property>
<!--  Preemption monitoring d Forehead time interval   Company :  millisecond  --><property>    <name>yarn.resourcemanager.monitor.preemption.monitoring_interval</name> <value>3000</value></property>
<!-- kill container The previous waiting time   Company :  millisecond  --><property>    <name>yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill</name> <value>15000</value></property>
<!--  The maximum value of resources preempted by each monitoring trigger （ Percentage of cluster resources ） --><property> <name>yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round</name> <value>0.1</value></property>
<!--  queue （ excess ） Percentage of resources that can be ignored without preemption  --><!--  That is, the queue has used more resources than capacity, But not more than (1+0.1)*capacity, Preempt the queue  --><property> <name>yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity</name> <value>0.1</value></property>

Besides , You can also set the queue separately , For example, specify a queue that does not allow preemption .

<!--  Queue preemption is prohibited  --><property>    <name>yarn.scheduler.capacity.$QUEUE_PATH.disable_preemption</name> <value>true</value></property>
<!--  Prohibit preemption in the queue  --><property>    <name>yarn.scheduler.capacity.$QUEUE_PATH.intra-queue-preemption.disable_preemption</name> <value>true</value></property>

【 Sorting out other problems 】

---

• The parent queue is set to preempt , When the sub queue has no configuration , What is the default ？

  A： When the sub queue has no configuration , It inherits the preemption attribute of the parent queue by default .

  
  

• The parent queue setting does not allow preemption , Sub queue settings allow preemption , Whether preemption is allowed for this sub queue ？

  A： The configuration of the sub queue shall prevail , Because the final task is submitted to the sub queue , That is, the actual use of resources is in the sub queue . Therefore, as long as the sub queue is configured to allow preemption , Even if the parent queue is set not to allow preemption , In fact, preemption will occur .

  
  

• What happens when the upper limit of resource adjustment when preemption is triggered each time still fails to meet the needs of other queues to be allocated ？

  A： Preemption will not take effect at this time .

  In the example above , The total resource of cluster is 18GB, Assume that the maximum resource size for each preemption is 0.1.

  That is to say, the maximum number of preemptions each time 18*1024*1024*0.1=1843.2MB, And single container need 3072MB Resources for , therefore , Even if all of them are preempted, it can't satisfy a single container Allocate the required resources , So the logic of preemption doesn't actually happen .

  
  

• Whether preemption within queues and preemption between queues can be set separately ？

  A： It can be set separately , They don't influence each other .
  

【 summary 】

---

To sum up , This paper describes the resource preemption in capacity scheduling , Including the configuration and use of resource preemption between queues and within queues , The principle of preemption 、 Related configuration 、 Using some scenes FAQ It also gives a brief description .

Okay , That's all for this article , Originality is not easy. , give the thumbs-up , Looking at , Sharing is the best support , thank you ～

Recommend relevant reading ：

---

YARN—— Correctly understand the importance of capacity scheduling capacity Parameters

YARN—— Several parameters that determine user resources in capacity scheduling

YARN—— Priority adjustment in the queue

This article is from WeChat official account. - hncscwc（gh_383bc7486c1a）.
If there is any infringement , Please contact the [email protected] Delete .
Participation of this paper “OSC Source creation plan ”, You are welcome to join us , share .