Spark's wide dependence and narrow dependence yyds dry goods inventory

Let's talk about wide dependence and narrow dependence

The core point to distinguish the wide and narrow dependence is Son RDD Of partition With the father RDD Of partition Whether it is 1 Relationship to many ,

If this is the case , Description multiple parents rdd Of partition Need to go through shuffle The process is summarized into a sub rdd Of partition, This is a wide dependency , stay DAGScheduler There will be stage The segmentation of .

Narrow dependence ：Narrow Dependency
Father RDD Hezi RDD Is a one-to-one dependency , Such as map,filter

Wide dependence ：Shuffle Dependency
Nature is shuffle. Such as reduceByKey,groupyByKey, Father RDD A partition data is given to the child RDD Multiple sections of

There is shuffle It's just wide dependence , Otherwise, it is narrow dependence

RDD As a data structure , It's essentially a Read only partition record set . One RDD Can contain multiple partitions , Each partition is a piece of dataset .

First , Narrow dependencies can be supported in The same node On , With pipeline Form to execute multiple commands （ Also called the same Stage The operation of ）, For example, in the implementation of map after , Followed by execution filter. contrary , Wide dependency requires that all parent partitions be available , You may need to call something like MapReduce And so on Cross node transfer .

secondly , From the perspective of failure recovery . Failure recovery with narrow dependency is more effective , Because it just needs to recalculate the lost parent partition that will do , And it can be recomputed on different nodes in parallel （ If a machine is too slow, it will be rescheduled to multiple nodes ）.