MapReduce (III)

One ：WritableComparable Sort

The order is MapReduce One of the most important operations in the framework .

MapTask and ReduceTask All data will be key Sort . The operation belongs to Hadoop Default behavior of . Data in any application is sorted , And whether or not it's logically necessary .

The default sort is in dictionary order , And the method to realize the sorting is Quick sort .

about MapTask, It will temporarily put the result of processing into the ring buffer , When the ring buffer makes

When the utilization reaches a certain threshold , Then do a quick sort of the data in the buffer , And put these ordinal numbers

Data overflow is written to disk , And when the data is processed , It will Merge and sort all files on disk .

about ReduceTask, It comes from every MapTask Remote copy of the corresponding data file on , If the file size exceeds a certain threshold , The overflow is written on the disk , Otherwise it's stored in memory . If the number of files on disk reaches a certain threshold , A merge sort is performed to generate a larger file ; If the size or number of files in memory exceeds a certain threshold , After a merge, overflow the data to disk . When all the data is copied ,ReduceTask Merge and sort all data on memory and disk at one time .

Sort and sort

（1） Partial sorting

MapReduce Sort the data set according to the key of the input record . Ensure the internal order of each output file .

（2） Total sort

There is only one file for the final output , And the documents are in order . The implementation is to set only one ReduceTask. But this method is used in

Very inefficient when dealing with large files , Because one machine processes all the files , Completely lost MapReduce The parallel architecture provided .

（3） Auxiliary sort ：（GroupingComparator grouping ） stay Reduce End to key Grouping . be applied to ： Receiving key by bean Object time , Want one or more fields to be the same （ All fields are different ） Of key Go to the same reduce When the method is used , You can use group sorting .

（4） Secondary sorting is in the process of custom sorting , If compareTo There are two judging conditions in, that is, the second order .

  Custom sort WritableComparable Principle analysis

bean Object as key transmission , Need to achieve WritableComparable Interface rewriting compareTo Method , You can sort .

WritableComparable Sorting case analysis （ Total sort ）

  Code implementation ：

（1）FlowBean Object in demand 1 The comparison function is added on the basis of

 （2）Mapper Class key by FlowBean,value by Text（ Phone number ）, No other code changes

（3） To write Reducer class

（4） To write Driver class

Two ：Combiner Merge

（1）Combiner yes MR In the program Mapper and Reducer A component other than .

（2）Combiner The parent class of a component is Reducer.

（3）Combiner and Reducer The difference is the location of the operation ,Combiner It's in every MapTask The node is running ;

（4）Combiner The meaning of this is for every MapTask Local summary of the output of , To reduce network traffic .

（5）Combiner The premise of application is not to affect the final business logic , and ,Combiner Output kv

It should be with Reducer The input of kv Types should be matched .

（6） Customize Combiner Implementation steps

  3、 ... and ：OutputFormat Data output