Resource scheduling and task scheduling of spark

Spark Resource scheduling and task scheduling

 

• Spark Resource scheduling and task scheduling process ：

After starting the cluster ,Worker The node will Master Node reports resources ,Master Master the cluster resources . When Spark To submit a Application after , according to RDD The dependency between will be Application To form a DAG Directed acyclic graph . After the task is submitted ,Spark Will be in Driver Create two objects at the end ：DAGScheduler and TaskScheduler,DAGScheduler It is a high-level scheduler for task scheduling , It's an object .DAGScheduler The main function of will be DAG according to RDD The wide and narrow dependencies are divided into Stage, And then put these Stage With TaskSet Form submitted to TaskScheduler（TaskScheduler It is a low-level scheduler for task scheduling , here TaskSet It's actually a collection , Inside the package is one by one task Mission , That is to say stage The degree of parallelism in task Mission ）,TaskSchedule Can traverse TaskSet aggregate , Get each task After the will task Send to compute node Executor To perform （ It's actually sent to Executor Thread pool in ThreadPool To carry out ）.task stay Executor The operation in the thread pool will be reported to TaskScheduler feedback , When task When execution fails , By TaskScheduler Responsible for retrying , take task Resend to Executor To carry out , Default retry 3 Time . If you try again 3 The first time still failed , So this task Where stage And failed .stage Failure is caused by DAGScheduler To try again , To resend TaskSet To TaskSchdeuler,Stage Default retry 4 Time . If you try again 4 Still fail after times , So this job And failed .job failed ,Application And failed .

TaskScheduler Can not only retry failed task, And try again straggling（ backward , slow ）task（ That is, the execution speed is faster than other task Too slow task）. If there are slow running task that TaskScheduler Will start a new task Come with this slow task Perform the same processing logic . Two task Which is the first to finish , Just one task The results of the implementation of . This is it. Spark Speculative execution mechanism . stay Spark Speculative execution is off by default . Speculative execution can be performed through spark.speculation Property to configure .

Be careful ：

• about ETL The type of business to be entered into the database needs to close the speculation execution mechanism , In this way, there will be no duplicate data warehousing .
• If you encounter data skew , If speculative execution is enabled, there may always be task Restart processing the same logic , The task may be in an endless state .

• The illustration Spark Resource scheduling and task scheduling process
• 

 

• Coarse grained resource requests and fine-grained resource requests

• Coarse grained resource requests (Spark）

stay Application Perform before , Apply all the resources , When the resource application is successful , The task will be scheduled , When all task After execution , This part of the resources will be released .

advantage ： stay Application Perform before , All the resources have been applied for , every last task Just use the resources directly , Unwanted task Apply for resources yourself before executing ,task It's going to start soon ,task Execution is fast ,stage Execution is fast ,job That's fast. ,application Execution is fast .

shortcoming ： Until the last one task Resources will only be released when the execution is completed , The resources of the cluster cannot be fully utilized .

• Fine grained resource requests （MapReduce）

Application You don't need to apply for resources before you execute , I'm going to do it directly , Give Way job Each of them task Apply for resources yourself before executing ,task Release resources when execution is complete .

advantage ： The resources of the cluster can be fully utilized .

shortcoming ：task Apply for resources yourself ,task Slow start ,Application The operation of the corresponding slow down .