当前位置:网站首页>JSON转MAP前后数据校验 -- 自定义UDF
JSON转MAP前后数据校验 -- 自定义UDF
2022-07-05 15:56:00 【牧码文】
JSON转MAP前后数据校验
JSON转MAP后的数据校验可以通过原始的sql实现,但是代码逻辑稍微复杂,为了简化操作,也可以通过自定义UDF实现。
因为想要实现的目的比较简单,就是map和json的转换前后比较,那么输入两个参数,在evalute方法中处理map和json即可,要做的也是逐个key比较value值即可
一、initialize方法
检查参数:initialize方法
@Override
public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumentException {
// 传入参数必须是两个
if (arguments.length != 2) {
throw new UDFArgumentLengthException("The function MAP_KEYS only accepts one argument.");
} else if (!(arguments[0] instanceof MapObjectInspector)) {
// 第一个参数类型需要是MAP
throw new UDFArgumentTypeException(0, "\""
+ ObjectInspector.Category.MAP.toString().toLowerCase()
+ "\" is expected at function JsonCheckMap, " + "but \""
+ arguments[0].getTypeName() + "\" is found");
} else if (!PrimitiveObjectInspector.PrimitiveCategory.STRING.equals(((PrimitiveObjectInspector)arguments[1]).getPrimitiveCategory())){
// 第二个参数类型需要是String
throw new UDFArgumentTypeException(0, "\""
+ PrimitiveObjectInspector.PrimitiveCategory.STRING.toString().toLowerCase()
+ "\" is expected at function JsonCheckMap, " + "but \""
+ arguments[1].getTypeName() + "\" is found");
}
mapOI = (MapObjectInspector) arguments[0];
ObjectInspector mapKeyOI = mapOI.getMapKeyObjectInspector();
// 返回结果为String类型,其实是个Boolean
return PrimitiveObjectInspectorFactory.javaStringObjectInspector;
}
二、evaluate方法
处理逻辑:evaluate方法
@Override
public Object evaluate(DeferredObject[] arguments) throws HiveException {
// 每次将集合置空
retArray.clear();
// 每次讲结果置true
isEquals = true;
Object mapObj = arguments[0].get();
String json = arguments[1].get().toString();
// 第二个参数转为json对象
JSONObject jsonObject = new JSONObject(json);
// 获得第一个参数的map
Map<?,?> mapVal = mapOI.getMap(mapObj);
// 遍历key
if (mapVal != null) {
retArray.addAll(mapVal.keySet());
for (int i = 0; i < retArray.size(); i++) {
try {
// 如果map中的key在json中不存在,返回false
String mapV = mapVal.get(retArray.get(i)).toString();
String jsonV = jsonObject.getString(retArray.get(i).toString());
if (!mapV.equals(jsonV)){
isEquals = false;
return false;
}
} catch (Exception e){
log.print(e.getMessage());
return false;
}
}
}
// 返回结果
return isEquals;
}
三、测试
打包上传集群,创建测试函数
测试1:
CREATE TEMPORARY FUNCTION cdp_to_json AS 'com.bilibili.udf.JsonCheckMap'
using jar "viewfs://jssz-bigdata-cluster/department/bigdata/file/hdfsfile/UDF.jar/UDF.jar";
select cdp_to_json(map('hello','world'), '{"hello":"wol"}')
false
测试2:
CREATE TEMPORARY FUNCTION cdp_to_json AS 'com.bilibili.udf.JsonCheckMap'
using jar "viewfs://jssz-bigdata-cluster/department/bigdata/file/hdfsfile/UDF.jar/UDF.jar";
select cdp_to_json(map('hello','world'), '{"hello":"world"}')
true
边栏推荐
猜你喜欢
Seaborn draws 11 histograms
【深度学习】深度学习如何影响运筹学?
2020-2022两周年创作纪念日
Today's sleep quality record 79 points
DeSci:去中心化科学是Web3.0的新趋势?
[Netease Yunxin] research and practice of super-resolution technology in the field of real-time audio and video
Cs231n notes (top) - applicable to 0 Foundation
Flet教程之 09 NavigationRail 基础入门(教程含源码)
Intel 13th generation Raptor Lake processor information exposure: more cores, larger cache
Use of set tag in SQL
随机推荐
Starkware: to build ZK "universe"
10 minutes to help you get ZABBIX monitoring platform alarm pushed to nail group
Cartoon: what is service fusing?
Win11提示无法安全下载软件怎么办?Win11无法安全下载软件
Use of RLOCK lock
单商户 V4.4,初心未变,实力依旧!
国泰君安网上开户安全吗
不敢买的思考
Single merchant v4.4 has the same original intention and strength!
Apiccloud cloud debugging solution
CISP-PTE之SQL注入(二次注入的应用)
Intel 13th generation Raptor Lake processor information exposure: more cores, larger cache
Cartoon: what is MapReduce?
[graduation season] as a sophomore majoring in planning, I have something to say
Flet教程之 12 Stack 重叠组建图文混合 基础入门(教程含源码)
sql中查询最近一条记录
《MongoDB入门教程》第04篇 MongoDB客户端
The new version of effect editor is online! 3D rendering, labeling, and animation, this time an editor is enough
Research and practice of super-resolution technology in the field of real-time audio and video
What is the difference between EDI license and ICP business license