Spark Learning Notes (III) -- basic knowledge of spark core

Catalog

Port number Description

spark Core module

spark core

spark SQL

spark streaming

MLlib

GraphX

spark Core components

driver

executor

Master&Worker

ApplicationMaster

Run the architecture

Submit task process

Yarn Client

Yarn Cluster

---

Port number Description

4040（ Calculation ）：Spark View the current Spark-shell Operation task ;

8080（ resources ）：Standalone In mode ,Spark Master Web Port number ;

7077：Spark Master Internal communication services ;

18080：Spark History server port number ;

8088：Hadoop YARN Check the operation of the task and the port number .

spark Core module

 spark SQL、spark Streaming、spark GraphX、spark MLlib

spark core

Spark Core Provides Spark The most basic and core functions .

Spark Other functions It's all in Spark Core On the basis of .

spark SQL

Spark SQL yes Spark Components used to manipulate structured data .

adopt Spark SQL, Users can use SQL or HQL（hive） To query data .

spark streaming

Spark Streaming yes Spark Platform for real-time data stream computing components .

spark streaming Provides a wealth of data processing API.

MLlib

MLlib yes Spark Provides a library of machine learning algorithms .

GraphX

GraphX yes Spark The framework and algorithm library provided by graph computing .

spark Core components

Spark The core of is a computing engine .

driver

driver yes spark Drive nodes , Used to perform spark In the task main Method , Responsible for the execution of the actual code .

Simple understanding ,driver Is the program that drives the entire application to run .

Mainly responsible for ：

（1） Convert user programs into jobs （job）;

（2） stay Executor Scheduling tasks between (task) ;

（3） track Executor Implementation of ;

（4） adopt UI Show how the query runs . 

executor

executor It's in a cluster Worker One of the JVM process , Responsible for Spark Run specific tasks in the job , Tasks are independent of each other .

Spark App launch ,Executor The nodes start at the same time , And exists in Spark The life cycle of the application .

If there is Executor The node has failed or crashed ,spark It will schedule the tasks on the faulty node to other Executor Continue to run on the node .

Executor Core functions ：

（1） Responsible for the operation of the composition Spark Application tasks , And return the result to the drive ;

（2） For user programs that require caching RDD Provide memory storage ;

（3）RDD Cache directly in Executor In process , Tasks can make full use of cached data to speed up operations at runtime .

Master&Worker

Master and Worker： there Master It's an entry cheng , Responsible for resource scheduling and allocation , Cluster monitoring, etc ;Worker It's also a process , One Worker Running on a server in the cluster ,Master Allocate resources Worker Parallel processing and calculation of data .

ApplicationMaster

ApplicationMaster A resource container used to request the resource scheduler to execute tasks Container, Run the user's own program tasks job, Monitor the execution of the entire task , Track the status of the entire task , Handle exceptions such as task failure .

Run the architecture

Driver： Manage job task scheduling in the whole cluster .

Executor： Responsible for the actual execution of tasks .

Submit task process

There are generally two ways of deployment execution ：Client and Cluster

The main difference between the two models is ：Driver The location of the running node of the program .Client Mode will be used for monitoring and scheduling Driver The module executes on the client .Cluster Mode will be used for monitoring and scheduling Driver Modules in Yarn Execution in a cluster .

Yarn Client

（1）Driver Run on the local machine that the task submitted ;

（2）Driver and ResourceManager Communications , Apply to start ApplicationMaster;

（3）ResourceManager Distribute Container, In the right place NodeManager Start the ApplicationMaster;

（4）ApplicationMaster Responsible for providing ResourceManager apply Executor Memory ;

（5）ResourceManager Distribute container;

（6）ApplicationMaster At the designated NodeManager Start the Executor process ;

（7）Executor towards Driver Reverse registration ;

（8）Executor When all registration is completed Driver Start execution ;

（9） perform main function , perform Action operator （Action Operator triggers a Job）, Divide according to wide dependency stage, Every stage Generate corresponding TaskSet, take task Distribute to each Executor On the implementation .

Yarn Cluster

（1）Driver Task submitted ;

（2）Driver and ResourceManager Communications , Apply to start ApplicationMaster;

（3）ResourceManager Distribute Container, In the right place NodeManager Start the ApplicationMaster, This ApplicationMaster Namely Driver;

（4）ApplicationMaster Responsible for providing ResourceManager apply Executor Memory ;

（5）ResourceManager Distribute container;

（6）ApplicationMaster At the designated NodeManager Start the Executor process ;

（7）Executor towards Driver Reverse registration ;

（8）Executor When all registration is completed Driver Start execution ;

（9） perform main function , perform Action operator （Action Operator triggers a Job）, Divide according to wide dependency stage, Every stage Generate corresponding TaskSet, take task Distribute to each Executor On the implementation .

This article is just a record of learning notes ！！！