Basic configuration and use of spark

Catalog

Content :

Spark Three models of

Local Pattern （ Local mode ）

Standalone Pattern

Yarn Pattern

Be careful

Use of official cases

api Use

summary ：

---

Content :

spark Configuration of the three modes and spark The basic use method

Spark Three models of

Local Pattern （ Local mode ）

local Set up Master Methods ：local（ Default to a thread ）,local[k]（ Specified number of threads ）,local[*]( Most used cpu Set thread ); The thread executing is Worker

To configure ：

local There is no need to modify the configuration file ,Spark After installation, you can use it directly local Model to calculate and analyze data

Standalone Pattern

Operation mode and hadoop Of resourcemanage Very similar .

The configuration file

Modify this file , If you don't change the name of his reference document

Modify the content ：

Add the name of the host , Follow hadoop The same is for group and cluster services

This file also needs to be modified , If nothing, change the reference file

Configure the content

This is designated to run on that machine master, And specify the port number

The next step is to send the configuration file to slaves Each host in

This file modification specifies java The running address of

History server

To add a historical server, you need to modify this configuration file and add the following contents

Be careful ： Take out the notes

As above, the configuration file needs to be modified

stay hdps Created in directory file , And start up hdfs

HA

Modify the configuration file

Add content

Appoint zookeeper Location of , And hdfs At the same time, comment out

To avoid conflict

Then distribute the configuration file

Yarn Pattern

effect ： You don't need to build Spark Cluster of

The configuration file ：

1.

modify hadoop in yarn Configuration file for

Modify the content ：

The function of this configuration is to turn off operations with excessive memory , Otherwise, when the computing memory exceeds a certain limit spark It will turn off automatically

2. modify

Modify the content ：

Be careful

Before using different modes, you should comment or delete the contents of other modes in the configuration file .

Use of official cases

Pay attention to each different mode ,master The content of is different , There will be some differences in other places

api Use

object WordCount {
  def main(args: Array[String]): Unit = {
    //WordCount Development 

    //local Pattern 
    // establish sparkConf object 
      // Set up spark Deployment environment for 
    val config = new SparkConf().setMaster("local[*]").setAppName("WordCount")


    // establish spark Context object 
    val sc=new SparkContext(config)
    // Read the file , Read line by line （ Local search  file:///xxxxx）
    val lines = sc.textFile("file:///opt/module/ha/spark/in")
    // Decompose the data into words one by one 
    val words = lines.flatMap(_.split(" "))
    // Transformation structure 
    val wordToOne = words.map((_, 1))
    // Group aggregation 
    val wordToSum = wordToOne.reduceByKey(_ + _)
    // Printout 
    println(wordToSum.collect())

  }
}

summary ：

Today I learned how to spark At the same time, the official use of spark Three modes of and their configuration files ,spark The configuration file of is relatively simple, but it is inconvenient that it cannot coexist , At the same time, the use method of his official operation is too complex to be easy to remember , It is estimated that you still need to check the document for later use . Next is in java It is also cumbersome when used on , But it is much simpler than the command line