3 environment construction -standalone

1 Standalone framework

2 Standalone Environment installation operation

3 test

4 Spark Application Architecture

5 General section

1 StandAlone Principle ?
Master and Worker Roles exist as independent processes , And make up Spark Runtime environment ( colony )
2 Spark The character is StandAlone Distribution in ？
Master role :Master process , Worker role :Worker process , Driver Roles and Executor role :
Run as a thread on Worker in
3 Standalone How to submit Spark application ？
bin/spark-submit --master spark://server:7077
4 4040\8080\18080 What are the differences ?
4040 It is the port bound when a single program is running, which can be used to view the operation of this task
8080: The default is StandAlone Next , Master role ( process ) Of WEB port , To view the current Master( colony ) The state of
18080: The default is the port of the history server , Because after each program is run ,4040 The port is logged off . In the future, if you want to look back at the running state of a program, you can go through history
Server view , The historical server runs stably for a long time , You can view the running process of the recorded program at any time .

5 Job\State\Task The relationship between ?
One Spark The program will be divided into several subtasks (Job) function , every last Job It will split into multiple State( Stage ) Come on
function , every last State There will be many Task( Threads ) To perform specific tasks