Distributed resource management and task scheduling framework yarn

stay Hadoop1.x in MapReduce yes Master/Slave structure .1 individual JobTracker Take more than one TaskTracker, We call it MRv1.

JobTracker The main function

Resource management

Task scheduling

T   askTracker The main function

Perform tasks , Respond to JobTracker command

Report the heartbeat

   The main problem

JobTracker A single point of failure , If it hangs up , The whole system doesn't work

JobTracker Too much load

Support only MR Computing framework , Suitable for batch processing 、 Disk based computing

There is no good decoupling design between resources and computing , A cluster can only use one computing framework

Yarn characteristic

Decoupling design of resource management and computing framework , A cluster resource is shared among the upper computing frameworks

Data sharing within the cluster is consistent , Data no longer needs to be copied and transferred between clusters , Achieve sharing and interoperability

Avoid single point of failure 、 Cluster resource expansion has been reasonably solved

Yarn Operation flow

ResourceManger( Resource management ):

ResourceScheduler( Resource scheduling )

AplicationsManger( Process management )

NodeManger Node resource management

ApplicationMaster task management

Task   Where the task runs

 

Client To the end ResourceManger Submit the assignment , These include applicationMaster Program , start-up applicationMaste The order of

    ResourceManger Assign the first Container, That is, allocate a container , And corresponding NodeManger signal communication , send NodeManger Start the job in this container ApplicationMaster

    NodeManger Start a Container function ApplicationMaster

ApplicationMaster First of all to REsourceManger register , So users can go through ResourceManger To query the running status of the job ,ApplicationMaster To ResourceManger Apply for various tasks and resources , And monitor the running status of the task

ApplicationMaster Get the resources and the corresponding NodeManger signal communication , Start the task

NodeManger receive ApplicationMaster command , start-up Contain Mission

each Container adopt RPC towards ApplicationMaster Report task status and progress , therefore ApplicationMaster You can restart the task when it fails

After job completion ,ApplicationMaster towards ResourceMangerous Apply for cancellation and close yourself

ResourceManger monitor NodeMangerous and ApplicationMaster

NodeManger Cyclical ResourceManger Report resource usage , And running state

ApplicationMaster Monitoring task , You can make NodeManger Restart the mission

What an application needs Container There are two main categories , as follows ：

（1） function ApplicationMaster Of Container： This is from ResourceManager（ To the internal resource scheduler ） Application and start up , When a user submits an application , A unique ApplicationMaster Resources needed ;

（2） To run all kinds of tasks Container： This is from ApplicationMaster towards ResourceManager Applied , And by the ApplicationMaster And NodeManager Communication to start .

The above two categories Container It may be on any node , Their positions are usually random , namely ApplicationMaster It may run on the same node as the tasks it manages .

Scheduling strategy

     FIFO   fifo

 Capacity Scheduler( Container scheduler )  General runway , Don't occupy

Fair Scheduler( Fair scheduler ) Allow others to occupy , But when you use it yourself , It will make others lose some data

Yarn shell Instructions

View version information ：yarn version

Use yarn Submission of orders jar package ：

yarn jar jarName mainClassPath -Dk1=v1 -Dk2=v2 inputPath outputPath

View all application List information ：yarn application -list

Kill the designated application, Use command ：yarn application kill app-id

see yarn Current resource usage of ：yarn top