spark architecture

1 Background

The spark-based development is based on the spark framework, doing business-level development on it, and understanding the basic architecture of spark

You can quickly master the skills of using spark

2 Architecture Diagram

3 Terminology explanation

3.1 Cluster Manager

It is the manager of cluster resources, which can be understood as the running base of spark, on which all code runs

Spark supports 3 cluster deployment modes: Standalone, Yarn, Mesos

3.2 Driver Program

The main method in the spark program

Created SparkContext, SparkContext sends Task to Executor for execution

Here is a point, the JVM of the Driver Program and the Executor do not share a JVM, not even on the same machine

3.3 Executor

The code area in the code where the operator is executed

Run on the worker node and execute the Task sent by the Driver

Returns the result of the operation to the Driver Manager