Data verification before and after JSON to map -- custom UDF

JSON turn MAP Before and after data verification

JSON turn MAP The data after verification can pass the original sql Realization , But the code logic is slightly complicated , To simplify the operation , You can also customize UDF Realization .

Because the purpose you want to achieve is relatively simple , Namely map and json Before and after the conversion , Then enter two parameters , stay evalute Method map and json that will do , What we have to do is one by one key Compare value value

One 、initialize Method

Inspection parameters ：initialize Method

@Override
    public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumentException {
    
      	//  The incoming parameters must be two 
        if (arguments.length != 2) {
    
            throw new UDFArgumentLengthException("The function MAP_KEYS only accepts one argument.");
        } else if (!(arguments[0] instanceof MapObjectInspector)) {
    
          //  The first parameter type needs to be MAP
            throw new UDFArgumentTypeException(0, "\""
                    + ObjectInspector.Category.MAP.toString().toLowerCase()
                    + "\" is expected at function JsonCheckMap, " + "but \""
                    + arguments[0].getTypeName() + "\" is found");
        } else if (!PrimitiveObjectInspector.PrimitiveCategory.STRING.equals(((PrimitiveObjectInspector)arguments[1]).getPrimitiveCategory())){
    
          //  The second parameter type needs to be String
            throw new UDFArgumentTypeException(0, "\""
                    + PrimitiveObjectInspector.PrimitiveCategory.STRING.toString().toLowerCase()
                    + "\" is expected at function JsonCheckMap, " + "but \""
                    + arguments[1].getTypeName() + "\" is found");
        }

        mapOI = (MapObjectInspector) arguments[0];
        ObjectInspector mapKeyOI = mapOI.getMapKeyObjectInspector();
        //  The return result is String type , Is actually a Boolean
        return PrimitiveObjectInspectorFactory.javaStringObjectInspector;
    }

Two 、evaluate Method

Processing logic ：evaluate Method

@Override
    public Object evaluate(DeferredObject[] arguments) throws HiveException {
    
        //  Set empty each time 
        retArray.clear();
      	//  Set the result every time true
        isEquals = true;
        Object mapObj = arguments[0].get();
        String json = arguments[1].get().toString();
      	//  The second parameter is changed to json object 
        JSONObject jsonObject = new JSONObject(json);
      	//  Get the first parameter map
        Map<?,?> mapVal = mapOI.getMap(mapObj);
      	//  Traverse key
        if (mapVal != null) {
    
            retArray.addAll(mapVal.keySet());
            for (int i = 0; i < retArray.size(); i++) {
    
                try {
    
                //  If map Medium key stay json Does not exist in the , return false
                String mapV = mapVal.get(retArray.get(i)).toString();
                String jsonV = jsonObject.getString(retArray.get(i).toString());
                if (!mapV.equals(jsonV)){
    
                    isEquals = false;
                    return false;
                }
                } catch (Exception e){
    
                    log.print(e.getMessage());
                    return false;
                }
            }
        }
        //  Return results 
        return isEquals;
    }

3、 ... and 、 test

Package and upload cluster , Create a test function

test 1:

CREATE TEMPORARY FUNCTION cdp_to_json AS 'com.bilibili.udf.JsonCheckMap'
using jar "viewfs://jssz-bigdata-cluster/department/bigdata/file/hdfsfile/UDF.jar/UDF.jar";
select cdp_to_json(map('hello','world'), '{"hello":"wol"}')

false

test 2:

CREATE TEMPORARY FUNCTION cdp_to_json AS 'com.bilibili.udf.JsonCheckMap'
using jar "viewfs://jssz-bigdata-cluster/department/bigdata/file/hdfsfile/UDF.jar/UDF.jar";
select cdp_to_json(map('hello','world'), '{"hello":"world"}')

true