Spark-day01- get started quickly

1：spark What is it?

Spark It is a memory based fast 、 Universal 、 Scalable big data analysis and calculation engine .

2：spark and Hadoop The difference between - Use scenarios

        Hadoop： One time data calculation , When the framework processes data , Will read data from the storage device , Perform logical operations , The result of the processing is then re stored in the media .

spark：spark and Hadoop The fundamental difference is the problem of data communication between multiple jobs ：spark Data communication between multiple jobs is memory based , and Hadoop It's disk based

3：spark Core module

spark core：spark The most basic and core functions

spark sql：spark Components used to manipulate structured data

spark streaming：spark Platform for real-time data stream computing components

spark mllib： Machine learning algorithm library

spark graphx： The framework and algorithm library provided by graph computing

4：word count Two ways to implement the case ---- The second is commonly used

package com.atguigu.bigdata.spark.wc

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

object Spark01_WordCount {
  def main(args: Array[String]): Unit = {
    //application
    //spark frame 
    // Establish and spark The connection of , The local environment 
    val sparConf = new SparkConf().setMaster("local").setAppName("WordCount")
    val sc = new SparkContext(sparConf);
    // Perform business operations 
    //1. Read the file , Get row by row data   "hello world"
    val lines:RDD[String] = sc.textFile("datas");
    //2. Split a row of data , Form words one by one  "hello word"=>hello,word
    val words:RDD[String] = lines.flatMap(_.split(" "));
    //3. Group data according to words , Easy to count  （hello,hello,hello）(world,world)
    val wordGroup:RDD[(String,Iterable[String])] = words.groupBy(word=>word);
    //4. Convert the grouped data   (hello,3) (world,2)
    val wordToCount = wordGroup.map{
      case (word,list) => {
        (word,list.size)
      }
    }
    //5. Acquisition output console 
    val array:Array[(String,Int)] = wordToCount.collect()
    array.foreach(println)
    // Close the connection 
    sc.stop();
  }
}

package com.atguigu.bigdata.spark.wc

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

object Spark02_WordCount {
  def main(args: Array[String]): Unit = {
    //application
    //spark frame 
    // Establish and spark The connection of , The local environment 
    val sparConf = new SparkConf().setMaster("local").setAppName("WordCount")
    val sc = new SparkContext(sparConf);
    // Perform business operations 
    //1. Read the file , Get row by row data   "hello world"
    val lines:RDD[String] = sc.textFile("datas");
    //2. Split a row of data , Form words one by one  "hello word"=>hello,word
    val words:RDD[String] = lines.flatMap(_.split(" "));

    //3. After each word , They are all marked with 1
    val wordToOne = words.map{
      word => (word,1)
    }
    //4. Group data according to words , Easy to count  （hello,hello,hello）(world,world)
    val groupRDD:RDD[(String,Iterable[(String,Int)])] = wordToOne.groupBy(
      t => t._1
    )
    //5. Convert the grouped data   (hello,3) (world,2)
    val wordToCount = groupRDD.map{
      case (word,list) => {
        list.reduce(
          (t1, t2) => {
            (t1._1, t1._2 + t2._2)
          }
        )
      }
    }
    //6. Acquisition output console 
    val array:Array[(String,Int)] = wordToCount.collect()
    array.foreach(println)
    // Close the connection 
    sc.stop();
  }
}

 5： Use spark function , Realization word count

package com.atguigu.bigdata.spark.wc

import org.apache.spark.rdd.RDD
import org.apache.spark.{SparkConf, SparkContext}

object Spark03_WordCount {
  def main(args: Array[String]): Unit = {
    //application
    //spark frame 
    // Establish and spark The connection of , The local environment 
    val sparConf = new SparkConf().setMaster("local").setAppName("WordCount")
    val sc = new SparkContext(sparConf);
    // Perform business operations 
    //1. Read the file , Get row by row data   "hello world"
    val lines:RDD[String] = sc.textFile("datas");
    //2. Split a row of data , Form words one by one  "hello word"=>hello,word
    val words:RDD[String] = lines.flatMap(_.split(" "));

    //3. After each word , They are all marked with 1
    val wordToOne = words.map{
      word => (word,1)
    }

    //spark The framework provides more functionality , Grouping and aggregation can be implemented using one method 
    // same key The data of , It can be done to value Conduct reduce polymerization 
    val wordToCount = wordToOne.reduceByKey((x,y) => {x+y})

    val array:Array[(String,Int)] = wordToCount.collect()
    array.foreach(println)
    // Close the connection 
    sc.stop();
  }
}