Spark practice 1: build spark operation environment in single node local mode

Preface ：

Spark For its own use scala Written , Running on the JVM above .

 JAVA edition :java 6 /higher edition.

1 download Spark

http://spark.apache.org/downloads.html

You can choose the version you need , My choice here is ：

http://d3kbcqa49mib13.cloudfront.net/spark-1.1.0-bin-hadoop1.tgz

If you are a good farmer who works hard , You can download the source code yourself ：http://github.com/apache/spark.

Be careful ： I'm running here Linux In the environment . You can install it on the virtual machine without conditions ！

2 decompression & Entry directory

tar -zvxf spark-1.1.0-bin-Hadoop1.tgz

cd spark-1.1.0-bin-hadoop1/

3 start-up shell

./bin/spark-shell

You will see a lot of things printed , Finally, it shows

4 A profound

Execute the following statements successively

val lines = sc.textFile("README.md")

lines.count()

lines.first()

val pythonLines = lines.filter(line => line.contains("Python"))

scala> lines.first() res0: String = ## Interactive Python Shel

--- explain , What is? sc

sc It is generated by default SparkContext object .

such as

 scala> sc res13: org.apache.spark.SparkContext = [email protected]

  Here is only local operation , Let's know in advance Distributed A diagram of the calculation ：

5  Independent program

Finally, I will conclude this section with an example

In order to make it run smoothly , Just follow the steps below ：

-------------- The directory structure is as follows ：

/usr/local/spark-1.1.0-bin-hadoop1/test$ find . . ./src ./src/main ./src/main/scala ./src/main/scala/example.scala ./simple.sbt

then simple.sbt Is as follows ：

name := "Simple Project" 

Copy

version := "1.0" 

Copy

scalaVersion := "2.10.4" 

Copy

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0"

Copy

 example.scala Is as follows ：

import org.apache.spark.SparkConf import org.apache.spark.SparkContext import org.apache.spark.SparkContext._

object example {   def main(args: Array[String]) {   val conf = new SparkConf().setMaster("local").setAppName("My App")   val sc = new SparkContext("local", "My App")   sc.stop()   //System.exit(0)   //sys.exit()   println("this system exit ok!!!")   } }

  Red local: One colony Of URL, Here is local, tell spark How to connect a colony ,local It means running in a single thread on the local machine without connecting to a cluster .

Orange My App: The name of a project ,

And then execute ：sbt package

  Execute after success

./bin/spark-submit  --class  "example"  ./target/scala-2.10/simple-project_2.10-1.0.jar

give the result as follows ：

It shows that the implementation is indeed successful ！

end ！