[untitled] user defined function

UDF(user defined function) Custom function 1：1

Customize a class

Inherit UDF class

rewrite evaluate Method

Put the project package To hdfs, Then imported into the hive

Add jar hdfs:///user/mazhichao/..jar

stay hive Can be used by creating temporary functions in

Create temporary function name as ‘ Package name . Method name ’

public class ValueMaskUDF extends UDF{

       public String evaluate(String input,int maxSaveStringLength,String replaceSign) {

             if(input.length()<=maxSaveStringLength){

                    return input;

             }

             return input.substring(0,maxSaveStringLength)+replaceSign;

       }

       public static void main(String[] args) {

             System.out.println(new ValueMaskUDF().evaluate(" Hebei Province ",2,"..."));

UDAF(user defined aggregation function) Custom aggregate functions  n:1

Inherit UDAF class

Internally define a static class , Realization UDAFEvaluator Interface

Implementation method init,iterate,terminatePartial,merge,terminate5 A way

The same is add jar etc.

public class DIYCountUDAF extends UDAF {  

    // Log object initialization , Make the visiting class have the ability to output logs 

    public static Logger logger=Logger.getLogger(DIYCountUDAF.class);

    

    // Static class implementation UDAFEvaluator

    public static class Evaluator implements UDAFEvaluator {  

        // Set member variables , Store the total number of records in each statistical range 

        private int totalRecords;  

        // Initialization function ,map and reduce Will execute this function , Play the role of initializing the variables required 

        public Evaluator() {  

            init();  

        }  

        // initialization , The initial value is 0, And log the corresponding output 

        public void init() {  

            totalRecords = 0;  

            logger.info("init totalRecords="+totalRecords);

        }  

        //map Stage , The return value is boolean type , When it comes to true Then the program continues , When it comes to false The program exits   

        public boolean iterate(String input) {

            // When input When the input is not empty , That is, there is a value , That is, it exists 1 That's ok , So do +1 operation 

            if (input != null) {  

                totalRecords += 1;  

            }  

            // Output the number of data processed by the current group 

            logger.info("iterate totalRecords="+totalRecords);

            return true;  

        }  

        /**

         *  Be similar to combiner, stay map Do partial aggregation within the scope , Pass the results on to merge Parameters in functions mapOutput  

         *  If you need to aggregate , On the other hand iterator Processing of returned results , Otherwise go straight back to iterator As a result of 

         */

        public int terminatePartial() {  

            logger.info("terminatePartial totalRecords="+totalRecords);

            return totalRecords;  

        }

        

        // reduce  Stage , For iterative processing map Each of them is different key Corresponding  terminatePartial Result 

        public boolean merge(int mapOutput) {  

            totalRecords +=mapOutput;  

            logger.info("merge totalRecords="+totalRecords);

            return true;  

        }  

        // Handle merge The result after calculation , At this time count stay merge Finish time , The results have come to , There is no need to deal with the overall results again , So just go back directly 

        public int terminate() {  

            logger.info("terminate totalRecords="+totalRecords);

            return totalRecords;  

        }  

    }  

}    

UDTF(User-Defined Table-Generating Functions)  Customize 1：n Table generation

Commonly used lateral view explode+udf To replace