Understanding: idea uses Scala to write wordcount programs and generate jar packages

preparation ：

At this time in your PC On , Yours JDK It should have been installed and configured JDK1.8 edition .

① First download on the official website IDEA( Suggest Ultimate edition ) https://www.jetbrains.com/idea/download/#section=windows

② from http://idea.lanyus.com/ Get the registration code or other ways to crack ( Personal student edition , Have not tested these websites )

③ open IDEA, install sbt and scala plug-in unit ,file-settings-Plugins Direct search scala and sbt Can be installed .

 

 

 

④ New project , choice scala->sbt;

Choose the right JDK and Scala edition ; The version here must correspond to ,JDK It is best to 1.8 edition ,Scala It's better not to be 2.12.x above , at present 2.12 And above should be arrived spark2.4 Version only （ To submit your homework spark The version of the cluster corresponds to ）.

 

 

Now the establishment project is completed .

Write programs and package ：

① To write scala Program ,src->main->scala Right click ,new->scala class;

choice Kind by Object;

Then enter your program .

package com.hq
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._

object wordcount {
  def main(args: Array[String]){
        if (args.length < 1) {
          System.err.println("Usage:")
          System.exit(1)
        }

    val conf = new SparkConf()
    // sc yes Spark Context, refer to “ Context ”, That is, the environment in which we run , Need to put conf When parameters are passed in ;
    val sc = new SparkContext(conf)
    // adopt sc Get one （hdfs Upper ） text file ,args（0） It is the parameter passed in from our console ,local If it runs, it will pass in a local text path
    val input = sc.textFile(args(0))

    // The following is wordcount Specific execution code 
    val lines=input.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_)

    lines.saveAsTextFile(args(1))

    sc.stop()
  }
}

 

② Set up build.sbt file

name:="Wordcount"

 

version:="0.1"

 

scalaVersion:="2.11.0"

 

libraryDependencies+="org.apache.spark"%%"spark-core"%"2.3.0"

③ pack ,File->Project Structure->Artifacts, spot “+”->JAR->From……;

 

choice module and mainclass,JAR files from libraries Select the second term ( You can choose the first item online , However, I hung up on the first item );

 

Check include in project build, spot ok;

then Build-Build Artifacts Generate jar package , If already META-INF Generated , Must be deleted before build.

So far, the packaging is complete .