Spark entry learning-2

1 Dependency

Wide dependencies: with shuffle
One partition of the parent RDD will be depended on by multiple partitions of the child RDD

Narrow dependencies: no shuffle
One partition of the parent RDD will only be depended on by one partition of the child RDD

Summary:
Narrow dependencies: parallelization + fault tolerance
WideDependency: perform stage division (the stage after shuffle needs to wait for shuffle to execute.

2 DAG && Stage

Spark's DAG: is the flow chart of spark task/program execution!
The beginning of DAG: from the creation of RDD
The end of DAG: to the end of Action
There are several DAGs in a Spark program by several Action operations

Stage: It is the stage divided by shuffle in DAG!
The latter stage can be executed only after the previous stage is executed.
Each task in the same stage can be executed in parallel without waiting!

3 Glossary

4 Job Submission Process