当前位置:网站首页>Data verification before and after JSON to map -- custom UDF
Data verification before and after JSON to map -- custom UDF
2022-07-05 16:39:00 【Mu code text】
JSON turn MAP Before and after data verification
Content catalog
JSON turn MAP The data after verification can pass the original sql Realization , But the code logic is slightly complicated , To simplify the operation , You can also customize UDF Realization .
Because the purpose you want to achieve is relatively simple , Namely map and json Before and after the conversion , Then enter two parameters , stay evalute Method map and json that will do , What we have to do is one by one key Compare value value
One 、initialize Method
Inspection parameters :initialize Method
@Override
public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumentException {
// The incoming parameters must be two
if (arguments.length != 2) {
throw new UDFArgumentLengthException("The function MAP_KEYS only accepts one argument.");
} else if (!(arguments[0] instanceof MapObjectInspector)) {
// The first parameter type needs to be MAP
throw new UDFArgumentTypeException(0, "\""
+ ObjectInspector.Category.MAP.toString().toLowerCase()
+ "\" is expected at function JsonCheckMap, " + "but \""
+ arguments[0].getTypeName() + "\" is found");
} else if (!PrimitiveObjectInspector.PrimitiveCategory.STRING.equals(((PrimitiveObjectInspector)arguments[1]).getPrimitiveCategory())){
// The second parameter type needs to be String
throw new UDFArgumentTypeException(0, "\""
+ PrimitiveObjectInspector.PrimitiveCategory.STRING.toString().toLowerCase()
+ "\" is expected at function JsonCheckMap, " + "but \""
+ arguments[1].getTypeName() + "\" is found");
}
mapOI = (MapObjectInspector) arguments[0];
ObjectInspector mapKeyOI = mapOI.getMapKeyObjectInspector();
// The return result is String type , Is actually a Boolean
return PrimitiveObjectInspectorFactory.javaStringObjectInspector;
}
Two 、evaluate Method
Processing logic :evaluate Method
@Override
public Object evaluate(DeferredObject[] arguments) throws HiveException {
// Set empty each time
retArray.clear();
// Set the result every time true
isEquals = true;
Object mapObj = arguments[0].get();
String json = arguments[1].get().toString();
// The second parameter is changed to json object
JSONObject jsonObject = new JSONObject(json);
// Get the first parameter map
Map<?,?> mapVal = mapOI.getMap(mapObj);
// Traverse key
if (mapVal != null) {
retArray.addAll(mapVal.keySet());
for (int i = 0; i < retArray.size(); i++) {
try {
// If map Medium key stay json Does not exist in the , return false
String mapV = mapVal.get(retArray.get(i)).toString();
String jsonV = jsonObject.getString(retArray.get(i).toString());
if (!mapV.equals(jsonV)){
isEquals = false;
return false;
}
} catch (Exception e){
log.print(e.getMessage());
return false;
}
}
}
// Return results
return isEquals;
}
3、 ... and 、 test
Package and upload cluster , Create a test function
test 1:
CREATE TEMPORARY FUNCTION cdp_to_json AS 'com.bilibili.udf.JsonCheckMap'
using jar "viewfs://jssz-bigdata-cluster/department/bigdata/file/hdfsfile/UDF.jar/UDF.jar";
select cdp_to_json(map('hello','world'), '{"hello":"wol"}')
false
test 2:
CREATE TEMPORARY FUNCTION cdp_to_json AS 'com.bilibili.udf.JsonCheckMap'
using jar "viewfs://jssz-bigdata-cluster/department/bigdata/file/hdfsfile/UDF.jar/UDF.jar";
select cdp_to_json(map('hello','world'), '{"hello":"world"}')
true
边栏推荐
- [echart] resize lodash 实现窗口缩放时图表自适应
- 數據訪問 - EntityFramework集成
- 面对新的挑战,成为更好的自己--进击的技术er
- Cartoon: what is blue-green deployment?
- Seaborn绘制11个柱状图
- 帮忙看看是什么问题可以吗?[ERROR] Could not execute SQL stateme
- Data Lake (XIV): spark and iceberg integrated query operation
- 极坐标扇图使用场景与功能详解
- The new version of effect editor is online! 3D rendering, labeling, and animation, this time an editor is enough
- 一些認知的思考
猜你喜欢
"21 days proficient in typescript-3" - install and build a typescript development environment md
Seaborn绘制11个柱状图
ES6 deep - ES6 class class
2020-2022 two-year anniversary of creation
How to set the WiFi password of the router on the computer
普洛斯数据中心发布DC Brain系统,科技赋能智慧化运营管理
Win11提示无法安全下载软件怎么办?Win11无法安全下载软件
单商户 V4.4,初心未变,实力依旧!
ES6 drill down - Async functions and symbol types
Desci: is decentralized science the new trend of Web3.0?
随机推荐
搜索 正排索引 和 倒排索引 区别
The new version of effect editor is online! 3D rendering, labeling, and animation, this time an editor is enough
中间表是如何被消灭的?
[echart] resize lodash 实现窗口缩放时图表自适应
Benji Banas membership pass holders' second quarter reward activities update list
Enterprise backup software Veritas NetBackup (NBU) 8.1.1 installation and deployment of server
Spring Festival Limited "forget trouble in the year of the ox" gift bag waiting for you to pick it up~
Pspnet | semantic segmentation and scene analysis
ES6 deep - ES6 class class
利用GrayLog告警功能实现钉钉群机器人定时工作提醒
一些认知的思考
Raspberry pie 4B installation pytorch1.11
详解SQL中Groupings Sets 语句的功能和底层实现逻辑
企业级备份软件Veritas NetBackup(NBU) 8.1.1服务端的安装部署
數據訪問 - EntityFramework集成
CISP-PTE之SQL注入(二次注入的应用)
ES6 drill down - ES6 generator function
二叉树相关OJ题
Android 隐私沙盒开发者预览版 3: 隐私安全和个性化体验全都要
树莓派4b安装Pytorch1.11