Yarn notes

• YARN(Yet Another Resource Negotiator), Is a general resource management system and call platform . Say it is universal , Because YARN Not just participation MapReduce Task scheduling , Will also be involved in Storm,Spark And other services .
• YRAN Mainly responsible for the cluster Memory 、CPU And how to allocate resources when applications apply for resources .HDFS Mainly responsible for the management of resources on the disk .
• YARN It doesn't care what application is requesting resources , I don't care what the resources are used for , It only cares about applications that request resources , Then help allocate resources .
• YARN Of Three components ：ResourceManager、NodeManager、ApplicationMaster.
• Analysis of official structure diagram

In this architecture diagram , from Machine angle Look at , There are two main roles ：ResourceManager and NodeManager.

ResourceManager It's through yarn-site.xml in <name>yarn.resourcemanager.hostname</name> This attribute is used to configure ; Each node in the cluster will run NodeManager The role of .

from Application perspective Look at , A role is required in the architecture diagram ：ApplicationMaster(AppMstr), because YARN It's universal , So the name of this role is also a general designation , For example MapReduce This character is called MRAppMstr.AppMstr Mainly responsible for the internal management responsibilities of an application , For example, to manage map/reduce Task execution status, etc . Every application will have one at run time AppMstr To lead the management tasks of the application .

• Whatever is in the machine AppMstr, Or is it mapTask、reduceTask And so on are based on Containers In the form of , This is to ensure that resources are isolated from each other and do not interfere with each other when running multiple applications on each machine , After running, the container will be released , And recycle resources .
• ResourceManager： It is the top-level role of resource allocation in the whole cluster , Determines the resource allocation among all applications in the cluster ; It is mainly used to accept jobs submitted by users , And pass NodeManager To allocate and manage computing resources on each node .
• ResourceScheduler： yes ResourceManager One of the internal components of ,ResourceManager Allocating resources is through ResourceScheduler To achieve .
• NodeManager： Running on each node , It is mainly responsible for managing the resource usage on this machine , For example, start the container 、 Release container 、 Recycling resources, etc ; towards ResourceManager Report all your resources 、 Used resources, etc , And receive ResourceManager The instruction assigned to it .
• ApplicationMaster： Occurs with the execution of the application . Only when the client submits the application , The application will run a corresponding AppMstr, Otherwise, if there is no application , And then there won't be AppMstr. It is internal to the application leader, In charge of the internal resource management and task execution of the application . When an application is submitted , The first process to run is AppMstr process .
• General process of job submission and operation ：

1. The client submits the job ,YARN Will be connected to ResourceManager.
2. After submitting the assignment ,YARN You will find an empty container on the machine and ask for it NodeManager function AppMstr（ Run first The procedure is AppMstr）, Run up AppMstr after ,AppMstr Will send to ResourceManager Register and maintain communication , In this way, we can see the task execution of the application on the console , For example, what percentage has been implemented . then AppMstr According to the tasks in your application （ such as mapTask、reduceTask Etc ） towards ResourceManager Application resources , And monitor the implementation of these tasks .
3. When AppMstr After applying for resources , Will require the corresponding NodeManager Start the container to execute the task .
4. Multiple tasks in an application may be running in multiple containers on multiple nodes , So these containers need to be directed to AppMstr Report on your implementation . For example MapReduce in ,mapTask/reduceTask Wait for multiple containers to AppMstr Container reporting . Each task is accomplished through RPC Agreement reporting status , And the user also passes RPC Agreement to AppMstr Query the execution of the current application .
5. NodeManager Regularly send to ResourceManager Report the resource usage and task execution on your own node .
6. When the application is finished ,AppMstr Will send to ResourceManager Log out of yourself and release the container .

• YARN Three schedulers in ：FIFO Scheduler、Capacity Scheduler、Fair Scheduler. By default Capacity Scheduler, You can also modify yarn-site.xml Medium <name>yarn.resourcemanager.scheduler.class</name> To configure other schedulers .