EasyExcel dynamic parsing and save table columns

背景

The data in one table is derived from exported tables from multiple other systems,The peculiarity is that most of the fields are the same（The exported headers may be different）,Only some of the few fields are unique to each system.Do a functional analysis around this

分析：Most fields are the same,Then there are the actual table fields,The only difference is that the names may vary from system to system,Few fields unique to each system,can be classified as dynamic fields.

总结：

• 公共字段（translation header：@ExcelProperty Multiple headers can be specified（ @ExcelProperty(value = {"发货数量", "采购数量(台)"}) ））
• 动态字段（There needs to be a corresponding relationship between the field name and the header of the dynamic field in each system,Consider using a dictionary,For salesman configuration,If you add other dynamic fields later, configure them directly in the dictionary,No further development is required）

注意：Since the actual header of the fixed field in the newly connected system cannot be controlled and expected,So if the public header of the new access system is inconsistent with the table fields,需要在 @ExcelProperty(value = {}) Add a new header to

效果

字典配置：

字典配置

数据表结果：

数据表结果

Common fields are stored using regular database table fields,Dynamic fields use extra column storage JSON 串.

代码

1. 引入pom坐标

<dependency>
	<groupId>com.alibaba</groupId>
	<artifactId>easyexcel</artifactId>
	<version>3.1.0</version>
</dependency>

复制

1. 创建实体类

public class AgentDeliverOrderImportVo {
    @ExcelProperty(value = {"订单编号"}, order = 1)
    private String deliverNo;

    @ExcelProperty(value = {"发货数量", "采购数量(台)"}, order = 14)
    @ColumnName(name = {"发货数量", "采购数量(台)"})
    private Integer deliverCount;
    
    /**
     * 动态字段（Business line number distinction）
     */
    private String dynamicFields;

    private Date createTime;

    private String createBy;
}

复制

1. Because there are indeterminate columns,所以只能使用 EasyExcel A write that does not create an object,那么

public String test(MultipartFile file) throws IOException {
    //Suppose you get dictionary values ​​from a dictionary
    Map<String, String> dictMap = new HashMap<>();
    dictMap.put("项目", "xm");
    dictMap.put("Hi pay for the order number", "hyfddbh");
    
    try (InputStream inputStream = file.getInputStream()) {
        EasyExcel.read(inputStream, new ReadListener<Map<String, String>>(){
            private Map<Integer, String> fieldHead;

            //获取表头
            @Override
            public void invokeHead(Map<Integer, ReadCellData<?>> headMap, AnalysisContext context) {
                Map<Integer, String> integerStringMap = ConverterUtils.convertToStringMap(headMap, context);
                log.info("解析到一条头数据:{}", JSON.toJSONString(integerStringMap));
                fieldHead = ExcelParsing.setFieldHead(integerStringMap, AgentDeliverOrderImportVo.class);
                log.info("Post-conversion header data：{}", JSONObject.toJSONString(fieldHead));
            }

            //获取数据
            @Override
            public void invoke(Map<String, String> map, AnalysisContext analysisContext) {
                log.info("解析到一条数据:{}", JSON.toJSONString(map));
                Map<String, String> valueMap = ExcelParsing.setFieldValue(fieldHead, dictMap, map);
                log.info("Convert a piece of data：{}", JSONObject.toJSONString(valueMap));
                log.info("Convert a piece of dynamic data：{}", JSONObject.toJSONString(ExcelParsing.getValueMap(valueMap, AgentDeliverOrderImportVo.class)));
            }

            @Override
            public void doAfterAllAnalysed(AnalysisContext analysisContext) {

            }
        }).sheet().doRead();
    }
    return "完成";
}

/**
 * @author Surpass
 * @Description: excel处理类
 * @date 27/07/2022 15:04
 */
class ExcelParsing {

    /**
     * Convert Chinese in public fields into database table fields,动态字段（其他字段保留）
     * @param headMap               {1:"姓名", 2:"年龄"}
     * @param obj               	AgentDeliverOrderImportVo(导入实体类)
     * @return java.util.Map<java.lang.String, java.lang.String>       {1:"name", 2:"年龄"}
     * @author Surpass
     * @date 01/08/2022 17:10
     */
    public static Map<Integer, String> setFieldHead(Map<Integer, String> headMap, Class<?> obj) {
        Field[] fields = obj.getDeclaredFields();
        for (Field field : fields) {
            ExcelProperty annotation = field.getAnnotation(ExcelProperty.class);
            if (annotation == null) {
                continue;
            }
            //There are cases where translation fields exist,A field corresponds to several headers（尽量避免）
            List<String> valueList = Arrays.asList(annotation.value());
            for (Map.Entry<Integer, String> entry : headMap.entrySet()) {
                if (valueList.contains(entry.getValue())) {
                    headMap.put(entry.getKey(), field.getName());
                }
            }
        }

        return headMap;
    }

    /**
     * 获取数据（平铺）,Refers to dynamic fieldskv和公共字段kv在同一级
     * @param headMap               {1:"name", 2:"年龄"}
     * @param dictMap               {"年龄":"age"}
     * @param valueMap              {1:"广州****公司", 2:"23"}
     * @return java.util.Map<java.lang.String, java.lang.String>
     * @author Surpass
     * @date 01/08/2022 17:10
     */
    public static Map<String, String> setFieldValue(Map<Integer, String> headMap,
                                                    Map<String, String> dictMap,
                                                    Map<String, String> valueMap) {
        Map<Integer, String> valueIntegerMap = valueMap.entrySet().stream().collect(
                Collectors.toMap(item -> Integer.valueOf(String.valueOf(item.getKey())),
                        item -> StrUtil.nullToEmpty(item.getValue()))
        );

        Map<String, String> valueResultMap = new HashMap<>(valueMap.size());
        Iterator<Map.Entry<Integer, String>> iterator = valueIntegerMap.entrySet().iterator();
        
        while (iterator.hasNext()) {
            Map.Entry<Integer, String> entry = iterator.next();
            //动态字段
            if (dictMap != null && dictMap.containsKey(headMap.get(entry.getKey()))) {
                valueResultMap.put(dictMap.get(headMap.get(entry.getKey())), entry.getValue());
                continue;
            }
            //公共字段
            valueResultMap.put(headMap.get(entry.getKey()), entry.getValue());
            iterator.remove();
        }
        return valueResultMap;
    }

    /**
     * 获取数据（表结构）,Refers to dynamic fieldskvhas been added to the database table field dynamicFields 中
     * @param obj               	AgentDeliverOrderImportVo(导入实体类)
     * @param valueMap              {"name":"广州****公司", "age":"23"}
     * @return java.util.Map<java.lang.String, java.lang.String>  
     * 返回结果： {"name":"广州****公司","dynamicFields":{"age":"23"}}
     * @author Surpass
     * @date 01/08/2022 17:10
     */
    public static Map<String, Object> getValueMap(Map<String, String> valueMap,
                                                  Class<?> obj) {
        Map<String, Object> resultMap = new HashMap<>(valueMap);
        List<String> commonFieldList = new ArrayList<>();
        Field[] fields = obj.getDeclaredFields();
        for (Field field : fields) {
            ExcelProperty annotation = field.getAnnotation(ExcelProperty.class);
            if (annotation == null) {
                continue;
            }
            commonFieldList.add(field.getName());
        }
        //Filter out public fields in entities
        Map<String, String> dynamicMap = valueMap.entrySet().stream()
                .filter(item -> !commonFieldList.contains(item.getKey()))
                .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
        resultMap.put("dynamicFields", dynamicMap);;
        return resultMap;
    }
}

复制

After parsing, the data of this document is consistent with the database table,Then our subsequent operation is the regular checksum insertion logic.

At present, there is a disadvantage that the dynamic fields stored in this way are not suitable for conditional queries,影响不是很大.

总结

本文介绍了使用 EasyExcel components to import,Implements the import of public column and dynamic column combination types,and how to store the function,Mainly use reflection and dictionary to maintain the correspondence between headers and fields of public columns and dynamic columns, respectively,Use this relationship to parse the data.