Spark practice case (upgraded version)

First step Prepare the environment first

1,IDEA Installed SDK yes 2.13.8 edition
2, newly build maven project , Depends on the following

  <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.13</artifactId>
      <version>3.3.0</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.13</artifactId>
      <version>3.3.0</version>
      <!--      <scope>provided</scope>-->
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.13</artifactId>
      <version>3.3.0</version>
      <scope>provided</scope>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-mllib -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-mllib_2.13</artifactId>
      <version>3.3.0</version>
      <scope>provided</scope>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive -->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-hive_2.13</artifactId>
      <version>3.3.0</version>
      <scope>provided</scope>
    </dependency>

Case a

Yes spark1.txt File filtering , take A Or include A Select the letters of and count them , Then output to dome1 In file .
The data are as follows

id	 Number 	 Content 
A	B	C
AB	A	B
C	A	B
AB	AB	AB

The code is as follows

def main(args: Array[String]): Unit = {
    
    var sparkConf = new SparkConf().setAppName("demo1").setMaster("local")
    var sc = new SparkContext(sparkConf)

    var filepath = "data/spark1.txt"
    // Get the first line 
    var fileFirst = sc.textFile(filepath).first()

    sc.textFile(filepath).filter(!_.equals(fileFirst)).flatMap(_.split("\t")).filter(_.contains("A"))
      .map((_,1)).reduceByKey(_+_).saveAsTextFile("data/demo1")

  }

The results are as follows

Case 2

Yes spark2 File filtering , Remove the first row of data, then count the total score and average score of each student's college entrance examination, and then output and save to dome2 In file .
Save format （ Examples ） The separator is \t
data

 full name 	 Chinese language and literature 	 mathematics 	 English 	 Synthesis 
 Zhang San 	90	89	100	120
 Li Si 	78	89	100	23
 Mei Mei Wang 	90	78	100	120

The code is as follows

def main(args: Array[String]): Unit = {
    
    var sparkConf = new SparkConf().setAppName("demo2").setMaster("local")
    var sc = new SparkContext(sparkConf)

    var filepath = "data/spark2.txt"

    var fileFirst = sc.textFile(filepath).first()
    sc.textFile(filepath).filter(!_.equals(fileFirst)).map(line=>{
    
      var arr = line.split("\t")
      var name = arr(0)
      var yw = arr(1).toInt
      var sx = arr(2).toInt
      var english = arr(3).toInt
      var lz = arr(4).toInt
      // Output 
      name+"\t"+(yw+sx+english+lz)+"\t"+(yw+sx+english+lz)/4
    }).saveAsTextFile("data/demo2")

  }

give the result as follows

Case three

 Read move.csv file , Remove the header of the first line , Change movie name , Duration , score , The data with null values in the four fields of release time will be removed , Then print out the number of data strips removed , And save the data to dome3 In file 

Save format ：
The data are as follows

 The movie name , Duration , score , Release time 
 The spring of the cattle herd ,97,8.9,2004-10-16
 The moon box of a journey to the West ,87,8.9,2014-10-24
 Smell and know women ,157,8.9,1992-12-23
 The valley of the wind ,117,8.9,
 Three stupid people make a lot of Bollywood ,171,8.9,2011-12-08
 The last emperor ,163,8.9,1987-10-23
 A brave heart ,177,8.9,1995-05-18
 The matrix 3： Matrix Revolution ,129,8.8,2003-11-05
 UP ,96,8.8,2009-08-04
 How to train your dragon master ,98,8.8,2010-05-14
 batman ： The dark knight ,152,9.0,2008-07-14
 The Lord of the rings 1： Ring protector ,178,9.0,2002-04-04
 Alive ,132,9.0,1994-05-17
 Saving Private Ryan ,169,8.9,1998-11-13
 The Lord of the rings 2： Twin towers ,179,9.0,2003-04-25
 The story of eight loyal dogs ,93,8.9,2009-06-13
 Legend of Shooting Heroes ,113,9.0,1993-02-05
 The past of America ,229,8.8,2015-04-23
 Disney's The Lion King ,89,9.0,1995-07-15
 The godfather 2,202,9.0,1974-12-12
 The Truman show ,103,9.0,
 The king of comedy ,85,9.5,1999-02-13
 Soul broken blue bridge ,108,9.5,1940-05-17
 Schindler's list ,195,9.5,1993-11-30
 Fast and furious 5,130,8.9,2011-05-12
 The sound of music ,174,9.0,1965-03-02
 Paradise cinema ,155,9.0,1988-11-17
 The city of the sky ,125,9.0,1992-05-01
 Havoc in heaven ,114,9.0,1965-12-31
 Wall-E ,98,9.0,2008-06-27
 Pirates of the Caribbean ,143,9.0,2003-11-21
 Edward Scissorhands ,105,9.0,1990-12-06
 The matrix ,136,9.0,2000-01-14
 The perfect world ,138,8.8,1993-11-24
 The Lord of the rings 3： The king is invincible ,201,9.0,2004-03-15
 Gone with the wind ,238,9.5,1939-12-15
 Titanic ,194,9.5,1998-04-03
 Infernal Affairs ,101,9.0,2003-09-05
 Harry · Porter and the Sorcerer's stone ,152,9.0,2002-01-26
 Shawshank redemption ,142,9.5,1994-09-10
 The Truman show ,103,9.0,
 Farewell my concubine ,171,9.5,1993-07-26
 Tang Baihu points autumn fragrance ,102,9.5,1993-07-01
 The king of comedy ,85,9.5,1999-02-13
 Disney's The Lion King ,89,9.0,1995-07-15
 Gone with the wind ,238,9.5,1939-12-15
 Titanic ,194,9.5,1998-04-03
 Roman holiday ,118,9.5,1953-08-20
 Léon ,110,9.5,1994-09-14
 Tang Baihu points autumn fragrance ,102,9.5,1993-07-01
 Seven Warriors ,207,8.8,1954-04-26
 Totoro ,86,9.1,2018-12-14
 Léon ,110,9.5,1994-09-14
 A Fei is passing ,94,9.1,2018-06-25
 Roman holiday ,118,9.5,1953-08-20
7 The present in the cell ,127,8.8,2013-01-23
 Shawshank redemption ,142,9.5,1994-09-10
 ocean ,104,9.1,2011-08-12
 Happy together ,96,9.0,1997-05-17
 Three gold escort ,161,9.1,1966-12-23
 "Avatar" ,162,8.9,2010-01-04
 Twelve Angry Man ,96,8.9,1957-04-13
 Fight club ,139,8.9,1999-09-10
 The pursuit of happyness ,117,8.9,2008-01-17
 heartache ,90,8.8,2010-07-26
 I love you! ,118,9.1,2011-02-17
 Brokeback Mountain ,134,8.8,2005-09-02
 Invincible Destruction King ,101,8.8,2012-11-06
 Love · get home ,80,9.1,2002-04-05
 Here comes the devil ,139,8.8,2000-05-13
 Hal's mobile castle ,119,8.9,2004-09-05
 Inception ,148,8.9,2010-09-01
 Sweet honey ,118,8.9,2015-02-13
 Princess ghost ,134,8.9,1998-05-01
 Harry · Porter and the Deathly Hallows （ Next ）,130,8.9,2011-08-04
 Time and space love travelers ,123,8.8,2013-09-04
 The godfather ,175,8.8,2015-04-18
 Bugging the storm ,137,8.8,2006-03-23
 The boy in striped pajamas ,94,8.8,2008-08-28
 Live horror ,97,8.8,2013-07-31
 Puerto Galera ,92,8.8,2009-07-31
 The city of God ,130,8.8,
 counsel ,127,8.8,2013-12-18
 True colour of a hero ,95,8.8,2017-11-17
 Deadly magic ,130,8.8,2006-10-17
 Farewell my concubine ,171,9.5,1993-07-26
 Migratory birds ,98,9.1,2001-12-12
 Beautiful life ,116,9.1,2020-01-03
 The Croods ,98,8.9,2013-04-20
 The little thing of first love ,118,8.9,2012-06-05
 Aliette, the little guy who borrowed things ,94,8.8,2010-07-17
 Despicable Me ,95,8.8,2010-06-20
 The fantasy drifting of youth school ,127,8.9,2012-11-22
V For vendetta ,132,8.9,2005-12-11
 batman ： The rise of the Dark Knight ,165,8.9,2012-08-27
 Marine pianist ,126,9.1,2019-11-15
 Spirited away ,125,9.1,2019-06-21
 The story of the loyal dog ,107,8.8,1987-08-01
 Beautiful mind ,135,8.8,2001-12-13
,173,8.8,2000-05-15

The code is as follows

def main(args: Array[String]): Unit = {
    
    var sparkConf = new SparkConf().setAppName("demo3").setMaster("local")
    var sc = new SparkContext(sparkConf)
    var filepath = "data/move.csv"
    // Get the first line 
    var fileFirst = sc.textFile(filepath).first()
    // Define the accumulation calculator 
    var longAccum = sc.longAccumulator("count")
    sc.textFile(filepath).filter(!_.equals(fileFirst)).filter(line=>{
    
      var arr = line.split(",")
      if(arr.length==4){
    
        for(i <- arr){
    
          if (i==""){
    
            longAccum.add(1)
            false
          }else{
    
            true
          }
        }
        true
      }else{
    
        longAccum.add(1)
        false
      }
    }).saveAsTextFile("data/demo3")
    println(" A total of ："+longAccum.value)
  }

Case study 4

 Yes film_log1.csv The file is processed with duplicate values , And print out the number of strips removed , Then save to film_log2 in 

The data are as follows

《 Up in the sky 》;2015.2.19;2015.3.29; Huanya film production Co., Ltd ; Ye Weixin , Zou Kaiguang ; Louis koo , Sammi Cheng , Wu Zhenyu , zhang , She Shiman , Guo Caijie ; The plot , love ; box office （ ten thousand ）1563.3; Beijing 
《 Hundred regiments war 》;2015.8.28;2015.10.11; Bayi film studio ; China Film Co., Ltd ; Beijing Forbidden City Film Company ; Ning Haiqiang , Zhang Yuzhong ; Tao zeru , Liu Zhibing , Yin Xiaotian , Wu yue , Tang Guoqiang , Wang Wufu ; The war / history ; box office （ ten thousand ）4137.3; tianjin 
《 Romantic weather 》;2015.10.23;2015.11.8;; Ning Ying ; Xia Yu , Guan Xiaotong , Qiu Ze ; romantic , love , comedy ; box office （ ten thousand ）75.2; Guangzhou 
《 Simple love 》;2015.7.3;2015.7.19; CTV Heli （ Beijing ） Cultural Investment Co., Ltd. Yiming film company （ The United States ）; Cui Lingyan ; Shao Yang Xu , Zhang Lin , Xie Yuqin , Shi Mingxi ; Urban romantic love comedy ; box office （ ten thousand ）232.7; Chengdu 
《 A thought of heaven 》;2015.12.31;2016.2.13; Tianhe feast , Kaideshengshi （ Beijing ） Investment Management Co., Ltd , And cloud chips ( Beijing ) Network technology co. LTD ; Zhang Cheng ; Shen Tang , Ma Li , Lin Xue , Du Xiaoyu , Prince , Li Yuanpeng ; comedy ; box office （ ten thousand ）829.5; shenyang 
《 The first experience of love 》;2015.8.7;2015.8.23; Shanghai Jinse Tianxia film and Television Co., Ltd ; ocean waves ; juck , Li Xiaofeng , Zhang Yao , Wu Dawei , Qu Jingjing , Liu Yaser , Qiao Xi ; comedy / love ; box office （ ten thousand ）31.7; Shanghai 
《 Hundred regiments war 》;2015.8.28;2015.10.11; Bayi film studio ; China Film Co., Ltd ; Beijing Forbidden City Film Company ; Ning Haiqiang , Zhang Yuzhong ; Tao zeru , Liu Zhibing , Yin Xiaotian , Wu yue , Tang Guoqiang , Wang Wufu ; The war / history ; box office （ ten thousand ）4137.3; jinan 
《 Purple xia 》;2015.12.11;2015.12.27; Yingcode company ; Miao Shuo ; Xu Jieer , Yuan Xiaochao , Liu Chengjun , The sky shines , Liu Yongjian , Tan he , Xiao Hong , Jinjing , Xie Yuanjiang ; love , fantasy ; box office （ ten thousand ）4.1; jinan 
《 Surprise all the way 》;2015.2.6;2015.3.8; Wanda Media Co., Ltd ; Jin Yimeng / Pan Anzi / Zhang Jiarui / Sondy ; Guo Caijie , Xiao Jingteng , Li-ying zhao , Feng Xiaoyue , Xia Yu , Ting Mei , Lan Yan , Lin Jiadong , Zhang Yi , ROC , Jiang Jinfu , Sun Yizhou , Xin Yuan Zhang , He Qingzi , Liu Wei , Joshua ; comedy / love / family ; box office （ ten thousand ）974.6; shenyang 
《 Loss of isolation 》;2015.3.20;2015.5.3; Huayi Brothers Media Group 、 Yuanhesheng film and television 、 Yingyi entertainment ; Peng Sanyuan ; Lau Andy , Jing bairan , Liang Jiahui , Wu Junru ; The plot 、 social ; box office （ ten thousand ）2167.9; Beijing 
《 Intruder 》;2015.4.30;2015.5.24; Winter and spring culture 、 Yinrun media 、 Herun media 、 Easy movie 、 Attractive film and television investment 、 Chongqing Film Group ; Wang Xiaoshuai ; Lu Zhong , Qin Hailu , Yuanzheng feng , Qin Hao , Pomegranate ; The plot 、 crime ; box office （ ten thousand ）103.6; Chengdu 
《 Loss of isolation 》;2015.3.20;2015.5.3; Huayi Brothers Media Group 、 Yuanhesheng film and television 、 Yingyi entertainment ; Peng Sanyuan ; Lau Andy , Jing bairan , Liang Jiahui , Wu Junru ; The plot 、 social ; box office （ ten thousand ）2167.9; fuzhou 
《 Simple love 》;2015.7.3;2015.7.19; CTV Heli （ Beijing ） Cultural Investment Co., Ltd. Yiming film company （ The United States ）; Cui Lingyan ; Shao Yang Xu , Zhang Lin , Xie Yuqin , Shi Mingxi ; Urban romantic love comedy ; box office （ ten thousand ）232.7; Guangzhou 
《 Detective files 》;2015.3.7;2015.3.22; One horse era cultural media （ Beijing ） Co., LTD. 、 Beijing Shengtang times Culture Communication Co., Ltd ; Peng fa ; Ma Yuke , Xin wu , Pan Yueming , Wang Jingchun , Mo Xiaoqi , Zhu Yuchen , Hong Tianming , Chen Guokun , Liu Yingyi ; The suspense , Thriller ; box office （ ten thousand ）34.1; shenyang 
《 Up in the sky 》;2015.2.19;2015.3.29; Huanya film production Co., Ltd ; Ye Weixin , Zou Kaiguang ; Louis koo , Sammi Cheng , Wu Zhenyu , zhang , She Shiman , Guo Caijie ; The plot , love ; box office （ ten thousand ）1563.3; shenyang 
《 Loss of isolation 》;2015.3.20;2015.5.3; Huayi Brothers Media Group 、 Yuanhesheng film and television 、 Yingyi entertainment ; Peng Sanyuan ; Lau Andy , Jing bairan , Liang Jiahui , Wu Junru ; The plot 、 social ; box office （ ten thousand ）2167.9; wuhan 
《 Bad guys must die 》;2015.11.27;2015.12.20; Beijing new power 、 Huayi Brothers 、 Nanjing Avenue Xingzhi ; SUN Hao ; Chen Bolin , Sun Yizhen , Qiao Zhenyu , Shenxianjun , Zhang Guang , Yang Xuwen , Ding Wenbo , Park Zhemin ; comedy , The suspense , travel , love ; box office （ ten thousand ）405.4; Changsha 
《 Hundred regiments war 》;2015.8.28;2015.10.11; Bayi film studio ; China Film Co., Ltd ; Beijing Forbidden City Film Company ; Ning Haiqiang , Zhang Yuzhong ; Tao zeru , Liu Zhibing , Yin Xiaotian , Wu yue , Tang Guoqiang , Wang Wufu ; The war / history ; box office （ ten thousand ）4137.3; Chengdu 
《 Everything grows 》;2015.4.17;2015.5.24; Beijing Lawley pictures 、 Hangzhou Guomai culture media 、 Beijing Lianrui film ; Li Yu ; Fan Bingbing , Han geng , Sha Yi , Wu Mochou , Yang Di , Qixi , Zhang Boyu , Shen Tingting , Li Meng , Ray Kesheng , Lu Xing ; love 、 The plot 、 campus 、 comedy ; box office （ ten thousand ）1431.8; Changsha 
《 Purple xia 》;2015.12.11;2015.12.27; Yingcode company ; Miao Shuo ; Xu Jieer , Yuan Xiaochao , Liu Chengjun , The sky shines , Liu Yongjian , Tan he , Xiao Hong , Jinjing , Xie Yuanjiang ; love , fantasy ; box office （ ten thousand ）4.1; Changsha 
《 Breaking wind 》;2015.8.7;2015.9.13; Evergrande film and Television Culture Co., Ltd ; Lin Chaoxian ; peng , Dou Xiao , Cui Shiyuan , Wang Luodan , Chen Jiale , Ouyang Nana , Lian Kai ; The plot 、 motion 、 love ; box office （ ten thousand ）1429.1; fuzhou 
《 Detective files 》;2015.3.7;2015.3.22; One horse era cultural media （ Beijing ） Co., LTD. 、 Beijing Shengtang times Culture Communication Co., Ltd ; Peng fa ; Ma Yuke , Xin wu , Pan Yueming , Wang Jingchun , Mo Xiaoqi , Zhu Yuchen , Hong Tianming , Chen Guokun , Liu Yingyi ; The suspense , Thriller ; box office （ ten thousand ）34.1; Guangzhou 
《 The most beautiful time to meet you 》;2015.12.11;2015.12.27; Guangzhou far-reaching culture Communication Co., Ltd ; Wu Na ; Tan songyun , Luo Yunxi ; campus , love , comedy ; box office （ ten thousand ）15.2; Beijing 
《 make the best of a bad bargain 》;2015.3.5;2015.3.29; China Film Co., Ltd ; Wang Ning ; Little Shenyang , Tian Liang , Jordan Chan , Xiong Dailin ; love 、 comedy ; box office （ ten thousand ）397.4; Shanghai 
《 The most beautiful time to meet you 》;2015.12.11;2015.12.27; Guangzhou far-reaching culture Communication Co., Ltd ; Wu Na ; Tan songyun , Luo Yunxi ; campus , love , comedy ; box office （ ten thousand ）15.2; shenyang 
《 Junior class 》;2015.6.19;2015.7.19; Gongfu film ; Huayi Brothers ; Xiao Yang ; Sun hong2 lei2 , Dongyu Zhou , Dong Zijian , Wang Lixin , Li Jiaqi , summer , Wang Sen ; youth 、 campus 、 comedy ; box office （ ten thousand ）506.7; Chengdu 
《 Say goodbye and I love you 》;2015.12.24;2016.1.17; Iqiyi pictures （ Beijing ） Co., LTD. 、 Sun Entertainment Culture Co., Ltd 、 Infinite Power Industrial Co., Ltd ; Ye Nianchen ; Fang Lishen , Deng Lixin ; love 、 The plot ; box office （ ten thousand ）173.2; fuzhou 
《 Since youth can't stay 》;2015.10.23;2015.11.22; Hangzhou Herun film and Television Co., Ltd ; Tian Meng ; Zhang Han , Chen Jon , Wang Xiaokun , Shi Yufei , Jia Shengqiang , Liao Juan ; comedy 、 love ; box office （ ten thousand ）500.3; Shanghai 
《 The former 2： Spare wheel counterattack 》;2015.11.6;2015.12.20; Huayi Brothers Media Co., Ltd 、 New temple pictures ; Tian Yusheng ; Zheng Kai , Guo Caijie , Zhang Yi Xing , Wang Zhuanjun ; love , comedy ; box office （ ten thousand ）2200.2; Guangzhou 
《 Everything grows 》;2015.4.17;2015.5.24; Beijing Lawley pictures 、 Hangzhou Guomai culture media 、 Beijing Lianrui film ; Li Yu ; Fan Bingbing , Han geng , Sha Yi , Wu Mochou , Yang Di , Qixi , Zhang Boyu , Shen Tingting , Li Meng , Ray Kesheng , Lu Xing ; love 、 The plot 、 campus 、 comedy ; box office （ ten thousand ）1431.8; Guangzhou 
《 Hundred regiments war 》;2015.8.28;2015.10.11; Bayi film studio ; Huayi Brothers Media Co., Ltd ; China Film Co., Ltd ; Beijing Forbidden City Film Company ; Ning Haiqiang , Zhang Yuzhong ; Tao zeru , Liu Zhibing , Yin Xiaotian , Wu yue , Tang Guoqiang , Wang Wufu ; The war / history ; box office （ ten thousand ）4137.3; jinan 
《 Simple love 》;2015.7.3;2015.7.19; CTV Heli （ Beijing ） Cultural Investment Co., Ltd. Yiming film company （ The United States ）; Cui Lingyan ; Shao Yang Xu , Zhang Lin , Xie Yuqin , Shi Mingxi ; Urban romantic love comedy ; box office （ ten thousand ）232.7; wuhan 
《 Surprise all the way 》;2015.2.6;2015.3.8; Wanda Media Co., Ltd ; Jin Yimeng / Pan Anzi / Zhang Jiarui / Sondy ; Guo Caijie , Xiao Jingteng , Li-ying zhao , Feng Xiaoyue , Xia Yu , Ting Mei , Lan Yan , Lin Jiadong , Zhang Yi , ROC , Jiang Jinfu , Sun Yizhou , Xin Yuan Zhang , He Qingzi , Liu Wei , Joshua ; comedy / love / family ; box office （ ten thousand ）974.6; jinan 
《 The first experience of love 》;2015.8.7;2015.8.23; Shanghai Jinse Tianxia film and Television Co., Ltd ; ocean waves ; juck , Li Xiaofeng , Zhang Yao , Wu Dawei , Qu Jingjing , Liu Yaser , Qiao Xi ; comedy / love ; box office （ ten thousand ）31.7; tianjin 
《 Breaking wind 》;2015.8.7;2015.9.13; Evergrande film and Television Culture Co., Ltd ; Lin Chaoxian ; peng , Dou Xiao , Cui Shiyuan , Wang Luodan , Chen Jiale , Ouyang Nana , Lian Kai ; The plot 、 motion 、 love ; box office （ ten thousand ）1429.1; shenyang 
《 Romantic weather 》;2015.10.23;2015.11.8;; Ning Ying ; Xia Yu , Guan Xiaotong , Qiu Ze ; romantic , love , comedy ; box office （ ten thousand ）75.2; Changsha 
《 Up in the sky 》;2015.2.19;2015.3.29; Huanya film production Co., Ltd ; Ye Weixin , Zou Kaiguang ; Louis koo , Sammi Cheng , Wu Zhenyu , zhang , She Shiman , Guo Caijie ; The plot , love ; box office （ ten thousand ）1563.3; tianjin 
《 Intruder 》;2015.4.30;2015.5.24; Winter and spring culture 、 Yinrun media 、 Herun media 、 Easy movie 、 Attractive film and television investment 、 Chongqing Film Group ; Wang Xiaoshuai ; Lu Zhong , Qin Hailu , Yuanzheng feng , Qin Hao , Pomegranate ; The plot 、 crime ; box office （ ten thousand ）103.6; shenyang 
《 make the best of a bad bargain 》;2015.3.5;2015.3.29; China Film Co., Ltd ; Wang Ning ; Little Shenyang , Tian Liang , Jordan Chan , Xiong Dailin ; love 、 comedy ; box office （ ten thousand ）397.4; wuhan 
《 Romantic weather 》;2015.10.23;2015.11.8;; Ning Ying ; Xia Yu , Guan Xiaotong , Qiu Ze ; romantic , love , comedy ; box office （ ten thousand ）75.2; Shanghai 
《 Simple love 》;2015.7.3;2015.7.19; CTV Heli （ Beijing ） Cultural Investment Co., Ltd. Yiming film company （ The United States ）; Cui Lingyan ; Shao Yang Xu , Zhang Lin , Xie Yuqin , Shi Mingxi ; Urban romantic love comedy ; box office （ ten thousand ）232.7; Changsha 
《 Romantic weather 》;2015.10.23;2015.11.8;; Ning Ying ; Xia Yu , Guan Xiaotong , Qiu Ze ; romantic , love , comedy ; box office （ ten thousand ）75.2; Shanghai 
《 The most beautiful time to meet you 》;2015.12.11;2015.12.27; Guangzhou far-reaching culture Communication Co., Ltd ; Wu Na ; Tan songyun , Luo Yunxi ; campus , love , comedy ; box office （ ten thousand ）15.2; shenyang 
《 make the best of a bad bargain 》;2015.3.5;2015.3.29; China Film Co., Ltd ; Wang Ning ; Little Shenyang , Tian Liang , Jordan Chan , Xiong Dailin ; love 、 comedy ; box office （ ten thousand ）397.4; Changsha 
《 Detective files 》;2015.3.7;2015.3.22; One horse era cultural media （ Beijing ） Co., LTD. 、 Beijing Shengtang times Culture Communication Co., Ltd ; Peng fa ; Ma Yuke , Xin wu , Pan Yueming , Wang Jingchun , Mo Xiaoqi , Zhu Yuchen , Hong Tianming , Chen Guokun , Liu Yingyi ; The suspense , Thriller ; box office （ ten thousand ）34.1; Changsha 
《 Purple xia 》;2015.12.11;2015.12.27; Yingcode company ; Miao Shuo ; Xu Jieer , Yuan Xiaochao , Liu Chengjun , The sky shines , Liu Yongjian , Tan he , Xiao Hong , Jinjing , Xie Yuanjiang ; love , fantasy ; box office （ ten thousand ）4.1; Beijing 
《 Loss of isolation 》;2015.3.20;2015.5.3; Huayi Brothers Media Group 、 Yuanhesheng film and television 、 Yingyi entertainment ; Peng Sanyuan ; Lau Andy , Jing bairan , Liang Jiahui , Wu Junru ; The plot 、 social ; box office （ ten thousand ）2167.9; fuzhou 
《 The most beautiful time to meet you 》;2015.12.11;2015.12.27; Guangzhou far-reaching culture Communication Co., Ltd ; Wu Na ; Tan songyun , Luo Yunxi ; campus , love , comedy ; box office （ ten thousand ）15.2; Changsha 
《 Love hairdresser 》;2015.7.17;2015.8.2; Beijing Renhe Bona culture media Co., Ltd ; Pour into the sea ; Travel , Zhang Ran , Mimi Chu , Huang Yifei , Cui Haobo , Wang Zixuan ; comedy  /  love ; box office （ ten thousand ）2.3; Shanghai 
《 A thought of heaven 》;2015.12.31;2016.2.13; Tianhe feast , Kaideshengshi （ Beijing ） Investment Management Co., Ltd , And cloud chips ( Beijing ) Network technology co. LTD ; Zhang Cheng ; Shen Tang , Ma Li , Lin Xue , Du Xiaoyu , Prince , Li Yuanpeng ; comedy ; box office （ ten thousand ）829.5; tianjin 
《 A mighty master 》;2015.2.19;2015.4.6; Yaolai culture , Huayi Brothers , Shanghai Film Group ; Lee Ren Kong ; Jackie Chan , John · Cusack , Adrian · Brody , Cui Shiyuan  , Lin Peng , Wang Ruoxin , Chopsticks brother , Simon son , William Feng , Zhujiayu ; action , Ancient costume , The plot , history ; box office （ ten thousand ）7443.2; Chengdu 
《 The most beautiful time to meet you 》;2015.12.11;2015.12.27; Guangzhou far-reaching culture Communication Co., Ltd ; Wu Na ; Tan songyun , Luo Yunxi ; campus , love , comedy ; box office （ ten thousand ）15.2; Beijing 
《 Everything grows 》;2015.4.17;2015.5.24; Beijing Lawley pictures 、 Hangzhou Guomai culture media 、 Beijing Lianrui film ; Li Yu ; Fan Bingbing , Han geng , Sha Yi , Wu Mochou , Yang Di , Qixi , Zhang Boyu , Shen Tingting , Li Meng , Ray Kesheng , Lu Xing ; love 、 The plot 、 campus 、 comedy ; box office （ ten thousand ）1431.8; Chengdu 
《 make the best of a bad bargain 》;2015.3.5;2015.3.29; China Film Co., Ltd ; Wang Ning ; Little Shenyang , Tian Liang , Jordan Chan , Xiong Dailin ; love 、 comedy ; box office （ ten thousand ）397.4; Guangzhou 
《 A mighty master 》;2015.2.19;2015.4.6; Yaolai culture , Huayi Brothers , Shanghai Film Group ; Lee Ren Kong ; Jackie Chan , John · Cusack , Adrian · Brody , Cui Shiyuan  , Lin Peng , Wang Ruoxin , Chopsticks brother , Simon son , William Feng , Zhujiayu ; action , Ancient costume , The plot , history ; box office （ ten thousand ）7443.2; Chengdu 
《 Intruder 》;2015.4.30;2015.5.24; Winter and spring culture 、 Yinrun media 、 Herun media 、 Easy movie 、 Attractive film and television investment 、 Chongqing Film Group ; Wang Xiaoshuai ; Lu Zhong , Qin Hailu , Yuanzheng feng , Qin Hao , Pomegranate ; The plot 、 crime ; box office （ ten thousand ）103.6; Changsha 
《 The first experience of love 》;2015.8.7;2015.8.23; Shanghai Jinse Tianxia film and Television Co., Ltd ; ocean waves ; juck , Li Xiaofeng , Zhang Yao , Wu Dawei , Qu Jingjing , Liu Yaser , Qiao Xi ; comedy / love ; box office （ ten thousand ）31.7; tianjin 
《 Breaking wind 》;2015.8.7;2015.9.13; Evergrande film and Television Culture Co., Ltd ; Lin Chaoxian ; peng , Dou Xiao , Cui Shiyuan , Wang Luodan , Chen Jiale , Ouyang Nana , Lian Kai ; The plot 、 motion 、 love ; box office （ ten thousand ）1429.1; fuzhou 
《 Since youth can't stay 》;2015.10.23;2015.11.22; Hangzhou Herun film and Television Co., Ltd ; Tian Meng ; Zhang Han , Chen Jon , Wang Xiaokun , Shi Yufei , Jia Shengqiang , Liao Juan ; comedy 、 love ; box office （ ten thousand ）500.3; tianjin 
《 Say goodbye and I love you 》;2015.12.24;2016.1.17; Iqiyi pictures （ Beijing ） Co., LTD. 、 Sun Entertainment Culture Co., Ltd 、 Infinite Power Industrial Co., Ltd ; Ye Nianchen ; Fang Lishen , Deng Lixin ; love 、 The plot ; box office （ ten thousand ）173.2; fuzhou 
《 Thumping star 》;2015.12.3;2016.1.10; Huanrui century , Jiaxing media , Youth light ; Chen Guohui ; Yang Mi , Li Yifeng , Chen Shu , Yaoqing wang , Delireba , Zhang Yunlong ; The city , love , comedy ; box office （ ten thousand ）1593.9; shenyang 
《 Surprise all the way 》;2015.2.6;2015.3.8; Wanda Media Co., Ltd ; Jin Yimeng / Pan Anzi / Zhang Jiarui / Sondy ; Guo Caijie , Xiao Jingteng , Li-ying zhao , Feng Xiaoyue , Xia Yu , Ting Mei , Lan Yan , Lin Jiadong , Zhang Yi , ROC , Jiang Jinfu , Sun Yizhou , Xin Yuan Zhang , He Qingzi , Liu Wei , Joshua ; comedy / love / family ; box office （ ten thousand ）974.6; jinan 
《 Detective files 》;2015.3.7;2015.3.22; One horse era cultural media （ Beijing ） Co., LTD. 、 Beijing Shengtang times Culture Communication Co., Ltd ; Peng fa ; Ma Yuke , Xin wu , Pan Yueming , Wang Jingchun , Mo Xiaoqi , Zhu Yuchen , Hong Tianming , Chen Guokun , Liu Yingyi ; The suspense , Thriller ; box office （ ten thousand ）34.1; Chengdu 
《 Up in the sky 》;2015.2.19;2015.3.29; Huanya film production Co., Ltd ; Ye Weixin , Zou Kaiguang ; Louis koo , Sammi Cheng , Wu Zhenyu , zhang , She Shiman , Guo Caijie ; The plot , love ; box office （ ten thousand ）1563.3; Chengdu 
《 The first experience of love 》;2015.8.7;2015.8.23; Shanghai Jinse Tianxia film and Television Co., Ltd ; ocean waves ; juck , Li Xiaofeng , Zhang Yao , Wu Dawei , Qu Jingjing , Liu Yaser , Qiao Xi ; comedy / love ; box office （ ten thousand ）31.7; Beijing 
《 Loss of isolation 》;2015.3.20;2015.5.3; Huayi Brothers Media Group 、 Yuanhesheng film and television 、 Yingyi entertainment ; Peng Sanyuan ; Lau Andy , Jing bairan , Liang Jiahui , Wu Junru ; The plot 、 social ; box office （ ten thousand ）2167.9; shenyang 
《 Junior class 》;2015.6.19;2015.7.19; Gongfu film ; Huayi Brothers ; Xiao Yang ; Sun hong2 lei2 , Dongyu Zhou , Dong Zijian , Wang Lixin , Li Jiaqi , summer , Wang Sen ; youth 、 campus 、 comedy ; box office （ ten thousand ）506.7; wuhan 
《 Romantic weather 》;2015.10.23;2015.11.8;; Ning Ying ; Xia Yu , Guan Xiaotong , Qiu Ze ; romantic , love , comedy ; box office （ ten thousand ）75.2; tianjin 
《 Up in the sky 》;2015.2.19;2015.3.29; Huanya film production Co., Ltd ; Ye Weixin , Zou Kaiguang ; Louis koo , Sammi Cheng , Wu Zhenyu , zhang , She Shiman , Guo Caijie ; The plot , love ; box office （ ten thousand ）1563.3; Changsha 
《 Everything grows 》;2015.4.17;2015.5.24; Beijing Lawley pictures 、 Hangzhou Guomai culture media 、 Beijing Lianrui film ; Li Yu ; Fan Bingbing , Han geng , Sha Yi , Wu Mochou , Yang Di , Qixi , Zhang Boyu , Shen Tingting , Li Meng , Ray Kesheng , Lu Xing ; love 、 The plot 、 campus 、 comedy ; box office （ ten thousand ）1431.8; tianjin 
《 A thought of heaven 》;2015.12.31;2016.2.13; Tianhe feast , Kaideshengshi （ Beijing ） Investment Management Co., Ltd , And cloud chips ( Beijing ) Network technology co. LTD ; Zhang Cheng ; Shen Tang , Ma Li , Lin Xue , Du Xiaoyu , Prince , Li Yuanpeng ; comedy ; box office （ ten thousand ）829.5; Chengdu 
《 Villain angel 》;2015.12.24;2016.2.13; Tianjin orange image media Co., Ltd 、 Beijing Guangguang Film Co., Ltd ; Deng Chao 、 Yu Baimei ; Deng Chao , Sun Li , "36229b; , Dailele ; comedy / absurd / love ; box office （ ten thousand ）6495.0; fuzhou 
《 Intruder 》;2015.4.30;2015.5.24; Winter and spring culture 、 Yinrun media 、 Herun media 、 Easy movie 、 Attractive film and television investment 、 Chongqing Film Group ; Wang Xiaoshuai ; Lu Zhong , Qin Hailu , Yuanzheng feng , Qin Hao , Pomegranate ; The plot 、 crime ; box office （ ten thousand ）103.6; tianjin 
《 The former 2： Spare wheel counterattack 》;2015.11.6;2015.12.20; Huayi Brothers Media Co., Ltd 、 New temple pictures ; Tian Yusheng ; Zheng Kai , Guo Caijie , Zhang Yi Xing , Wang Zhuanjun ; love , comedy ; box office （ ten thousand ）2200.2; Shanghai 
《 A mighty master 》;2015.2.19;2015.4.6; Yaolai culture , Huayi Brothers , Shanghai Film Group ; Lee Ren Kong ; Jackie Chan , John · Cusack , Adrian · Brody , Cui Shiyuan  , Lin Peng , Wang Ruoxin , Chopsticks brother , Simon son , William Feng , Zhujiayu ; action , Ancient costume , The plot , history ; box office （ ten thousand ）7443.2; Changsha 
《 Surprise all the way 》;2015.2.6;2015.3.8; Wanda Media Co., Ltd ; Jin Yimeng / Pan Anzi / Zhang Jiarui / Sondy ; Guo Caijie , Xiao Jingteng , Li-ying zhao , Feng Xiaoyue , Xia Yu , Ting Mei , Lan Yan , Lin Jiadong , Zhang Yi , ROC , Jiang Jinfu , Sun Yizhou , Xin Yuan Zhang , He Qingzi , Liu Wei , Joshua ; comedy / love / family ; box office （ ten thousand ）974.6; Beijing 
《 Villain angel 》;2015.12.24;2016.2.13; Tianjin orange image media Co., Ltd 、 Beijing Guangguang Film Co., Ltd ; Deng Chao 、 Yu Baimei ; Deng Chao , Sun Li , "36229b; , Dailele ; comedy / absurd / love ; box office （ ten thousand ）6495.0; shenyang 
《 Intruder 》;2015.4.30;2015.5.24; Winter and spring culture 、 Yinrun media 、 Herun media 、 Easy movie 、 Attractive film and television investment 、 Chongqing Film Group ; Wang Xiaoshuai ; Lu Zhong , Qin Hailu , Yuanzheng feng , Qin Hao , Pomegranate ; The plot 、 crime ; box office （ ten thousand ）103.6; Guangzhou 
《 The former 2： Spare wheel counterattack 》;2015.11.6;2015.12.20; Huayi Brothers Media Co., Ltd 、 New temple pictures ; Tian Yusheng ; Zheng Kai , Guo Caijie , Zhang Yi Xing , Wang Zhuanjun ; love , comedy ; box office （ ten thousand ）2200.2; tianjin 
《 Junior class 》;2015.6.19;2015.7.19; Gongfu film ; Huayi Brothers ; Xiao Yang ; Sun hong2 lei2 , Dongyu Zhou , Dong Zijian , Wang Lixin , Li Jiaqi , summer , Wang Sen ; youth 、 campus 、 comedy ; box office （ ten thousand ）506.7; Chengdu 
《 Purple xia 》;2015.12.11;2015.12.27; Yingcode company ; Miao Shuo ; Xu Jieer , Yuan Xiaochao , Liu Chengjun , The sky shines , Liu Yongjian , Tan he , Xiao Hong , Jinjing , Xie Yuanjiang ; love , fantasy ; box office （ ten thousand ）4.1; tianjin 
《 The first experience of love 》;2015.8.7;2015.8.23; Shanghai Jinse Tianxia film and Television Co., Ltd ; ocean waves ; juck , Li Xiaofeng , Zhang Yao , Wu Dawei , Qu Jingjing , Liu Yaser , Qiao Xi ; comedy / love ; box office （ ten thousand ）31.7; Guangzhou 
《 Everything grows 》;2015.4.17;2015.5.24; Beijing Lawley pictures 、 Hangzhou Guomai culture media 、 Beijing Lianrui film ; Li Yu ; Fan Bingbing , Han geng , Sha Yi , Wu Mochou , Yang Di , Qixi , Zhang Boyu , Shen Tingting , Li Meng , Ray Kesheng , Lu Xing ; love 、 The plot 、 campus 、 comedy ; box office （ ten thousand ）1431.8; jinan 
《 Breaking wind 》;2015.8.7;2015.9.13; Evergrande film and Television Culture Co., Ltd ; Lin Chaoxian ; peng , Dou Xiao , Cui Shiyuan , Wang Luodan , Chen Jiale , Ouyang Nana , Lian Kai ; The plot 、 motion 、 love ; box office （ ten thousand ）1429.1; fuzhou 
《 Love hairdresser 》;2015.7.17;2015.8.2; Beijing Renhe Bona culture media Co., Ltd ; Pour into the sea ; Travel , Zhang Ran , Mimi Chu , Huang Yifei , Cui Haobo , Wang Zixuan ; comedy  /  love ; box office （ ten thousand ）2.3; Beijing 
《 Simple love 》;2015.7.3;2015.7.19; CTV Heli （ Beijing ） Cultural Investment Co., Ltd. Yiming film company （ The United States ）; Cui Lingyan ; Shao Yang Xu , Zhang Lin , Xie Yuqin , Shi Mingxi ; Urban romantic love comedy ; box office （ ten thousand ）232.7; Chengdu 
《 Since youth can't stay 》;2015.10.23;2015.11.22; Hangzhou Herun film and Television Co., Ltd ; Tian Meng ; Zhang Han , Chen Jon , Wang Xiaokun , Shi Yufei , Jia Shengqiang , Liao Juan ; comedy 、 love ; box office （ ten thousand ）500.3; jinan 
《 Junior class 》;2015.6.19;2015.7.19; Gongfu film ; Huayi Brothers ; Xiao Yang ; Sun hong2 lei2 , Dongyu Zhou , Dong Zijian , Wang Lixin , Li Jiaqi , summer , Wang Sen ; youth 、 campus 、 comedy ; box office （ ten thousand ）506.7; Chengdu 
《 The most beautiful time to meet you 》;2015.12.11;2014.12.27; Guangzhou far-reaching culture Communication Co., Ltd ; Wu Na ; Tan songyun , Luo Yunxi ; campus , love , comedy ; box office （ ten thousand ）15.2; Changsha 
《 Say goodbye and I love you 》;2015.12.24;2016.1.17; Iqiyi pictures （ Beijing ） Co., LTD. 、 Sun Entertainment Culture Co., Ltd 、 Infinite Power Industrial Co., Ltd ; Ye Nianchen ; Fang Lishen , Deng Lixin ; love 、 The plot ; box office （ ten thousand ）173.2; jinan 
《 make the best of a bad bargain 》;2015.3.5;2015.3.29; China Film Co., Ltd ; Wang Ning ; Little Shenyang , Tian Liang , Jordan Chan , Xiong Dailin ; love 、 comedy ; box office （ ten thousand ）397.4; wuhan 
《 Romantic weather 》;2015.10.23;2015.11.8;; Ning Ying ; Xia Yu , Guan Xiaotong , Qiu Ze ; romantic , love , comedy ; box office （ ten thousand ）75.2; Chengdu 
《 Breaking wind 》;2015.8.7;2015.9.13; Evergrande film and Television Culture Co., Ltd ; Lin Chaoxian ; peng , Dou Xiao , Cui Shiyuan , Wang Luodan , Chen Jiale , Ouyang Nana , Lian Kai ; The plot 、 motion 、 love ; box office （ ten thousand ）1429.1; Beijing 
《 Purple xia 》;2015.12.11;2015.12.27; Yingcode company ; Miao Shuo ; Xu Jieer , Yuan Xiaochao , Liu Chengjun , The sky shines , Liu Yongjian , Tan he , Xiao Hong , Jinjing , Xie Yuanjiang ; love , fantasy ; box office （ ten thousand ）4.1; shenyang 
《 Love hairdresser 》;2015.7.17;2015.8.2; Beijing Renhe Bona culture media Co., Ltd ; Pour into the sea ; Travel , Zhang Ran , Mimi Chu , Huang Yifei , Cui Haobo , Wang Zixuan ; comedy  /  love ; box office （ ten thousand ）2.3; shenyang 
《 Thumping star 》;2015.12.3;2016.1.10; Huanrui century , Jiaxing media , Youth light ; Chen Guohui ; Yang Mi , Li Yifeng , Chen Shu , Yaoqing wang , Delireba , Zhang Yunlong ; The city , love , comedy ; box office （ ten thousand ）1593.9; fuzhou 
《 Say goodbye and I love you 》;2015.12.24;2016.1.17; Iqiyi pictures （ Beijing ） Co., LTD. 、 Sun Entertainment Culture Co., Ltd 、 Infinite Power Industrial Co., Ltd ; Ye Nianchen ; Fang Lishen , Deng Lixin ; love 、 The plot ; box office （ ten thousand ）173.2; shenyang 
《 The first experience of love 》;2015.8.7;2015.8.23; Shanghai Jinse Tianxia film and Television Co., Ltd ; ocean waves ; juck , Li Xiaofeng , Zhang Yao , Wu Dawei , Qu Jingjing , Liu Yaser , Qiao Xi ; comedy / love ; box office （ ten thousand ）31.7; Guangzhou 
《 Say goodbye and I love you 》;2015.12.24;2016.1.17; Iqiyi pictures （ Beijing ） Co., LTD. 、 Sun Entertainment Culture Co., Ltd 、 Infinite Power Industrial Co., Ltd ; Ye Nianchen ; Fang Lishen , Deng Lixin ; love 、 The plot ; box office （ ten thousand ）173.2; tianjin 
《 A thought of heaven 》;2015.12.31;2016.2.13; Tianhe feast , Kaideshengshi （ Beijing ） Investment Management Co., Ltd , And cloud chips ( Beijing ) Network technology co. LTD ; Zhang Cheng ; Shen Tang , Ma Li , Lin Xue , Du Xiaoyu , Prince , Li Yuanpeng ; comedy ; box office （ ten thousand ）829.5; fuzhou 

The code is as follows

def main(args: Array[String]): Unit = {
    
    var sparkConf = new SparkConf().setAppName("demo4").setMaster("local")
    var sc = new SparkContext(sparkConf)
    var filepath = "data/film_log1.csv"

    // Get the total number of entries 
    var move_count = sc.textFile(filepath).map((_,1)).reduceByKey(_+_).count()
    // Find the total quantity 
    var move_sum = sc.textFile(filepath).map((_,1)).reduceByKey(_+_).map(x=>x._2).sum()
    // Save the data 
    sc.textFile(filepath).map((_,1)).reduceByKey(_+_).map(x=>x._1).saveAsTextFile("data/demo4")

    println(" The total number of repetitions is p:"+(move_sum-move_count))
  }

Case study 5

stay film_log2 On the basis of documents , Handle missing values , There will be no director's information used " nothing " fill , And print out how many pieces of data are filled , Then save to film_log3 in
The code is as follows

def main(args: Array[String]): Unit = {
    
    var sparkConf = new SparkConf().setAppName("demo5").setMaster("local")
    var sc = new SparkContext(sparkConf)
    var filepath = "data/film_log1.csv"

    var  longAccum = sc.longAccumulator("count")

    sc.textFile(filepath).map(line=>{
    
      var arr = line.split(";")
      if (arr(4)==""){
    
        arr(4)=" nothing "
        var str=""
        for (i <- arr){
    
          str+=i+";"
        }
        str=str.substring(0,str.length-1)
        longAccum.add(1)
        str
      }else{
    
        line
      }

    }).saveAsTextFile("data/demo5")
    println(" A total of :"+longAccum.value)
  }

Case study 6

 stay film_log3 Change the date format to yyyy-MM-dd Format , Then save to film_log4 in 

def main(args: Array[String]): Unit = {
    
    var sparkConf = new SparkConf().setAppName("demo6").setMaster("local")
    var sc = new SparkContext(sparkConf)
    var filepath = "data/film_log1.csv"

    sc.textFile(filepath).map(line=>{
    
      var arr = line.split(";")
      arr(1)=arr(1).replace(".","-")
      arr(2)=arr(2).replace(".","-")
      var str = ""
      for (i <- arr){
    
        str+=i+";"
      }
      str=str.substring(0,str.length-1)
      // Output 
      str
    }).saveAsTextFile("data/demo6")
  }