Hadoop learning (3) - Yarn

YARN yes hadoop Resource scheduling platform of .

1. YARN Basic architecture

YARN It consists of the following components .Resource Manager、Node Manager、Application Master and Container form .

1. Resource Manager ：RM Is a global Explorer . Responsible for resource management and task scheduling of the whole system . The core component consists of two . Scheduler Scheduler  and Application task manager ASM（Applications Master）

Scheduler Responsible for allocating resources to applications .ASM Responsible for starting 、 monitor AM（Application Master）

1. Node Manager:
   1. Periodic direction RM Report node resources and Container Running state .
   2. receive AM The order of . start-up 、 stop it Container.
2. Application Master: Each task submitted by the user contains a AM.AM The main functions of the ：
   1. Segmentation data .
   2. For applications to RM Application resources （container）
   3. And NM signal communication , start-up 、 Stop task
   4. Monitor the running status of tasks . Provide fault tolerance mechanism .

 

1. Container:

Container It's the abstraction of resources （ Memory and CPU）.

2.  Job submission process

The user to YARN After submitting the task ,YARN Running the application in two phases . start-up AM and AM Create application 、 Request resources for the task 、 Monitor the task execution process until the task is completed .

 

establish AM

1)  Client to YARN Submit the application , And ask for RM Allocate resources

2) RM To start a Container, And run on it AM.

AM Create application

3) AM towards RM Medium ASM register .

4) AM towards RM Apply for and access resources .

5) AM After obtaining resources , Will correspond to the resource NM Establishing a connection .

6)  Start the task , Corresponding Container Will be in AM Registered at .

7)  Each task is directed to AM Periodic reporting on performance .

 

8) AM Run back to RM Log off and close yourself

3. 【 Need to improve 】 Resource scheduler

HADOOP There are three job schedulers for . First in, first out scheduler FIFO, Capacity scheduler （capacity scheduler） And fair scheduler （fair scheduler）. The default is capacity scheduler .

 

(1) FIFO ： Queue save job, First come, first go .

(2) Capacity Scheduler：

Support multiple queues , Each queue adopts a FIFO scheduling strategy .

(3) Fair Scheduler：