Yarn scheduling

1.*Yarn******** Of ******** Scheduler Scheduler*

Ideally , We apply to Yarn Requests for resources should be met immediately , But in reality, resources are often limited , Especially in a very busy cluster , An application resource request often needs to wait for a period of time to get to the corresponding resource . stay ****Yarn in , The one responsible for allocating resources to applications is Scheduler****. In fact, scheduling itself is a problem , It's hard to find a perfect strategy to solve all the application scenarios . So ,Yarn A variety of schedulers and configurable policies are available for us to choose from .

stay Yarn There are three schedulers to choose from ：FIFO Scheduler ,Capacity Scheduler,Fair Scheduler.

1.1.*FIFO Scheduler*

*FIFO* Scheduler Queue applications in the order they are submitted , This is a **** fifo **** queue , In resource allocation , First, allocate resources to the application at the top of the queue , Wait for the top application requirements to be met before the next allocation , And so on .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-W5mwaGl2-1613039337551)(file:///C:\Users\14112\AppData\Local\Temp\ksohtml15760\wps1.jpg)]

FIFO Scheduler Is the simplest and easiest to understand scheduler , It doesn't need any configuration , But it doesn't apply to shared clusters . Large applications may take up all cluster resources , This causes other applications to be blocked . In a Shared cluster , More suitable for Capacity Scheduler or Fair Scheduler, Both of these schedulers allow large and small tasks to obtain certain system resources while submitting .

1.2.*Capacity Scheduler*

Capacity The scheduler allows multiple organizations to share the entire cluster , Each organization can get part of the computing power of the cluster . By assigning a dedicated queue to each organization , And then allocate a certain cluster resource for each queue , In this way, the whole cluster can provide services to multiple organizations by setting multiple queues . besides , The queue can be divided vertically , In this way, multiple members within an organization can share the queue resource , Inside a queue , Resource scheduling is based on FIFO (FIFO) Strategy .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-RvqIKZZQ-1613039337554)(file:///C:\Users\14112\AppData\Local\Temp\ksohtml15760\wps2.jpg)]

Capacity scheduler Capacity Scheduler It was originally made by Yahoo The initial development design made Hadoop Applications can be used by multiple users , And maximize the throughput of the whole cluster resources , Present quilt IBM BigInsights and Hortonworks HDP Adopted by .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-k7hFoYd1-1613039337555)(file:///C:\Users\14112\AppData\Local\Temp\ksohtml15760\wps3.png)]

Capacity Scheduler Designed to allow applications to share cluster resources in a predictable and simple way , namely " Job queue ".Capacity Scheduler It is to allocate the existing resources to the running applications according to the needs and requirements of the tenants .Capacity Scheduler It also allows applications to access resources that are not yet in use , To ensure that the queues share the resources that other queues are allowed to use . Administrators can control the capacity of each queue ,Capacity Scheduler Responsible for submitting jobs to the queue .

1.3.*Fair Scheduler*

stay Fair In scheduler , We do not need to occupy certain system resources in advance ,Fair The scheduler will run for all job Dynamically adjust system resources . As shown in the figure below , When the first big job When submitting , This is the only one job Running , At this point it gets all the cluster resources ; When the second small task is submitted ,Fair The scheduler will allocate half the resources to this small task , Let these two tasks share cluster resources fairly .

It should be noted that , Below Fair In scheduler , There will be a delay from the second task submission to resource acquisition , Because it needs to wait for the first task to release the occupied Container. After small tasks are executed, they will also release the resources they occupy , The big task gets all the system resources . The end result is Fair The scheduler can not only achieve high resource utilization, but also ensure that small tasks can be completed in time .

[ Failed to transfer the external chain picture , The origin station may have anti-theft chain mechanism , It is suggested to save the pictures and upload them directly (img-4Xi6uyg6-1613039337558)(file:///C:\Users\14112\AppData\Local\Temp\ksohtml15760\wps4.jpg)]

Fair scheduler Fair Scheduler It was originally made by Facebook Development design makes Hadoop The application can share the whole cluster resources fairly by multiple users , Present quilt Cloudera CDH Adopted by .

Multiple users share the entire cluster resources fairly , Present quilt Cloudera CDH Adopted by .

Fair Scheduler There is no need to keep the resources of the cluster , Because it dynamically balances resources among all running jobs .