The Spark run on Yarn Spark application

Running Spark on Yarn provides the tightest integration with other Hadoop components and the easiest way to use Spark on an existing Hadoop cluster.For running Spark applications on Yarn, Spark provides two deployment modes: Client mode and Cluster mode.The Driver in Client mode runs on the client, while the Driver in Cluster mode runs on Yarn's Application Master.

Client mode must be used for programs with any interactive components (eg, spark-shell, pyspark).Client mode is also useful when building Spark applications, since any debug output is immediately visible.Cluster mode is suitable for production jobs because the entire application runs on the cluster, which makes it easier to preserve log files (including those from the Driver) for future exception checking.

1. Deployment Mode

In YARN, each application instance has an ApplicationMaster process, which is the first container started for that application.The application is responsible for requesting resources from the ResourceManager.Once the resource is allocated, the application will instruct the NodeManagers to start the container.ApplicationMasters remove the dependency on active clients: the process that started the application can terminate and continue to run cooperatively from a process managed by YARN on the cluster.

See Spark Application Deployment Tool spark-submit for options to specify the deployment mode.

1.1 Client Deployment Mode

In Client mode, the Spark Driver runs on the host that submits the job.App