Share an example of a simple MapReduce method using a virtual machine

I am also a new comer in big data , The bad news I share may only disturb the younger newcomers , Excuse me .

（1） First, the last mapreduce Flow chart of the method .

This is a method of counting the number of occurrences of text words , Very much in hadoop The wisdom of dividing and ruling the ecosystem , In the picture above  splitting and mapping Phases can run on different hosts or virtual machines , If it's distributed , It will run very fast .reducing The stage is to count the same words into a module , Finally, together . The implementation method on the virtual machine will be introduced later .

（2） Create... On the virtual machine .txt file

If there is no ready-made data set , Just use what I wrote above , Let's write the text into a .txt In file . Make sure that the virtual machine is configured hadoop environment variable ,hdfs and yarn Various configuration files have been edited , No error will be reported when running .

[[email protected] ~]# cd /opt/rh/hadoop-3.2.2/
[[email protected] hadoop-3.2.2]# ls
bin  etc  hdfs  include  lib  libexec  LICENSE.txt  logs  NOTICE.txt  README.txt  sbin  share  tmp  wcinput  wcoutput
[[email protected] hadoop-3.2.2]# mkdir wordcountfile
[[email protected] hadoop-3.2.2]# ls
bin  etc  hdfs  include  lib  libexec  LICENSE.txt  logs  NOTICE.txt  README.txt  sbin  share  tmp  wcinput  wcoutput  wordcountfile
[[email protected] hadoop-3.2.2]# cd wordcountfile
[[email protected] wordcountfile]# vi wordfile.txt
[[email protected] wordcountfile]# cat wordfile.txt
i like python
python like spark
Spark link hadoop
[[email protected] wordcountfile]#

So here we are hadoop The main file directory is created ,wordfile.txt file . Next we type in the command , function mapreduce Method .

（3） Open the virtual machine cluster ,start-all.sh, call hadoop Under the mapreduce Methods wordcount jar package , Specify the path to save the result file , Count the number of times each word appears .

[[email protected] hadoop-3.2.2]# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.2.jar wordcount  file:///opt/rh/hadoop-3.2.2/wordcountfile/wordfile.txt file:///opt/rh/hadoop-3.2.2/wordoutputfile

The virtual machine firewall needs to be turned off ,systemctl stop firewalld.service, The existence of firewall will affect the communication between hosts in the cluster . If you do not specify the full path of the output input file , It will be reported that the file has not been uploaded to hdfs Error of . Click on the run .

2021-04-03 21:54:23,391 INFO mapreduce.Job:  map 0% reduce 0%
2021-04-03 21:54:29,850 INFO mapreduce.Job:  map 100% reduce 0%
2021-04-03 21:54:38,973 INFO mapreduce.Job:  map 100% reduce 100%
2021-04-03 21:54:40,002 INFO mapreduce.Job: Job job_1617457620069_0002 completed successfully
2021-04-03 21:54:40,169 INFO mapreduce.Job: Counters: 54

See this note wordcount The program runs successfully . Let's look at the result file .

drwxr-xr-x. 2 root root     26 Apr  3 21:33 wordcountfile
drwxr-xr-x. 2 root root     88 Apr  3 21:58 wordoutputfile
[[email protected] hadoop-3.2.2]# cd wordoutputfile/
[[email protected] wordoutputfile]# ls
part-r-00000  _SUCCESS

  We turn on part-r-00000 file , See our results .

[[email protected] wordoutputfile]# cat part-r-00000
Spark   1
hadoop  1
i       1
like    2
link    1
python  2
spark   1
[[email protected] wordoutputfile]#

The result is the same as that in the above figure . Errors may be reported during execution .

But don't give up , If you encounter any problems, go online to search , You can also ask me . Although I don't know how to .