Easyexcel realizes dynamic import and export

Preface

YuQue community EasyExcel Import and export , When you know which headers are needed . Often the actual situation , A table has many fields , Some field users don't need that much , He just wants to see the key parts . Suppose a customer asks ： I don't need so many columns , My data doesn't have so many header items , I want to save time , What should I do ？ It can only be implemented dynamically , I don't know what column the user will fill in , Just receive , And then back to the user . Don't talk much , Direct code implementation .

Code implementation

springboot The construction of the project is omitted , Used SQL、 For configuration and dependency, please refer to what I wrote before ：EasyExcel3.0.5 Export multiple sheet, Including query optimization , Here is a detailed account , This article omits the tedious steps , Enter the core content .

Create two new core tables

We want to implement dynamic import , You need to use these two tables to deal with . One is the import result table , The other is the import result details , Why do you do this ？

Ideas ：
Because each user can import different content , Therefore, each import can use a unique batch as the result of the user's execution , Another user leads another batch . Each batch has N Multiple data , Is the imported details . When users want to query the contents of their own files , You can import data details by batch number Association . Look at the specific data table design ~~~

Import result table ： tb_import_result

CREATE TABLE `tb_import_result` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT ' Primary key ',
  `batch_number` varchar(50) NOT NULL COMMENT ' Import batch number ',
  `file_name` varchar(100) DEFAULT NULL COMMENT ' File name ',
  `upload_time` datetime DEFAULT NULL COMMENT ' Upload time ',
  `total` bigint(20) DEFAULT NULL COMMENT ' total ',
  `headers` varchar(4000) DEFAULT NULL COMMENT ' Header ',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;

Import result schedule ：tb_import_result_detail
The parts list contains the complete fields of the data table , And batch number （batch_number）. But users with so many fields don't need them all .

CREATE TABLE `tb_import_result_detail` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT ' Primary key ',
  `batch_number` varchar(100) COLLATE utf8_bin DEFAULT NULL COMMENT ' Import batch number ',
  `total_pay` bigint(20) DEFAULT NULL COMMENT ' Total sum , In minutes ',
  `actual_pay` bigint(20) DEFAULT NULL COMMENT ' Amount paid . Company : branch . Such as :20007, Express :200 element 7 branch ',
  `payment_type` tinyint(1) unsigned zerofill DEFAULT NULL COMMENT ' Payment type ,1、 Pay online ,2、 Cash on Delivery ',
  `post_fee` bigint(20) DEFAULT NULL COMMENT ' Postage . Company : branch . Such as :20007, Express :200 element 7 branch ',
  `create_time` datetime DEFAULT NULL COMMENT ' Order creation time ',
  `shipping_name` varchar(20) COLLATE utf8_bin DEFAULT NULL COMMENT ' Logistics name ',
  `shipping_code` varchar(20) COLLATE utf8_bin DEFAULT NULL COMMENT ' Logistics order No ',
  `buyer_message` varchar(100) COLLATE utf8_bin DEFAULT NULL COMMENT ' Buyer message ',
  `buyer_nick` varchar(50) COLLATE utf8_bin DEFAULT NULL COMMENT ' Buyer's nickname ',
  `buyer_rate` tinyint(1) DEFAULT NULL COMMENT ' Has the buyer commented on ,0 Not evaluated ,1 Evaluated ',
  `receiver_state` varchar(100) COLLATE utf8_bin DEFAULT '' COMMENT ' Harvest address （ province ）',
  `receiver_city` varchar(255) COLLATE utf8_bin DEFAULT '' COMMENT ' Harvest address （ City ）',
  `receiver_district` varchar(255) COLLATE utf8_bin DEFAULT '' COMMENT ' Harvest address （ District / county ）',
  `receiver_address` varchar(255) COLLATE utf8_bin DEFAULT '' COMMENT ' Harvest address （ The street 、 Address and other detailed addresses ）',
  `receiver_mobile` varchar(12) COLLATE utf8_bin DEFAULT NULL COMMENT ' Receiver's mobile phone ',
  `receiver_zip` varchar(15) COLLATE utf8_bin DEFAULT NULL COMMENT ' Post code of consignee ',
  `receiver` varchar(50) COLLATE utf8_bin DEFAULT NULL COMMENT ' Consignee ',
  `invoice_type` int(1) DEFAULT '0' COMMENT ' Invoice type (0 No invoice 1 Ordinary invoice ,2 Electronic invoice ,3 VAT invoice )',
  `source_type` int(1) DEFAULT '2' COMMENT ' Source of order ：1:app End ,2：pc End ,3：M End ,4： Wechat end ,5： mobile phone qq End ',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;

EasyExcel Dynamic import 、 export

The so-called dynamic import , No matter what header you import （ The premise is that some fields in a table in your database , There are many fields , The content imported by each user is different ）, The backend only needs to save the header and data imported by the user each time . Open the door ！

Controller layer

package cn.com.easyExcel.controller;

import cn.com.easyExcel.param.OrderExportParam;
import cn.com.easyExcel.service.OrderExcelService;
import cn.com.easyExcel.vo.ResultVo;
import io.swagger.annotations.Api;
import io.swagger.annotations.ApiOperation;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

@Api(tags = " Import and export order information ")
@RestController
@RequestMapping(value = "/order/excel")
public class OrderExcelController {
    

    @Autowired
    private OrderExcelService orderExcelService;

    @ApiOperation(value = " Dynamic import ")
    @PostMapping(value="/dynamicImport")
    public ResultVo<Void> dynamicImportExcel(@RequestParam(name = "file") MultipartFile file) throws IOException {
    
        orderExcelService.dynamicImportExcel(file);
        return ResultVo.successMsg(" Successful import ");
    }

    @ApiOperation(value = " Dynamic export ")
    @PostMapping(value="/dynamicExport")
    public void dynamicExportExcel(@RequestBody OrderExportParam param, HttpServletResponse response) throws IOException {
    
        orderExcelService.dynamicExportExcel(param, response);
    }
}

Service Realization

The complete code of dynamic export is all in this implementation class , Dynamic import has another Monitor , The key part , It will be explained later .

package cn.com.easyExcel.service.impl;

import cn.com.easyExcel.excel.converter.CellWriteWeight;
import cn.com.easyExcel.excel.listener.OrderImportListener;
import cn.com.easyExcel.excel.util.EasyExcelUtils;
import cn.com.easyExcel.mapper.ImportResultDetailMapper;
import cn.com.easyExcel.mapper.ImportResultMapper;
import cn.com.easyExcel.param.OrderExportParam;
import cn.com.easyExcel.pojo.ExcelHeader;
import cn.com.easyExcel.pojo.ImportResult;
import cn.com.easyExcel.pojo.ImportResultDetail;
import cn.com.easyExcel.pojo.OrderRsp;
import cn.com.easyExcel.service.ImportResultService;
import cn.com.easyExcel.service.OrderExcelService;
import cn.hutool.core.util.IdUtil;
import com.alibaba.excel.EasyExcel;
import com.alibaba.excel.EasyExcelFactory;
import com.alibaba.excel.ExcelWriter;
import com.alibaba.excel.write.metadata.WriteSheet;
import com.alibaba.fastjson.JSON;
import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;
import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.collections4.CollectionUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;

import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.List;

@Slf4j
@Service
public class OrderExcelServiceImpl implements OrderExcelService {
    

    @Autowired
    private ImportResultService importResultService;

    @Autowired
    private ImportResultDetailMapper detailMapper;

    @Autowired
    private ImportResultMapper resultMapper;

    private static final int PAGE_SIZE = 100;

    private static final String CONTENT_TYPE = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";

    private static final String CONTENT_DISPOSITION = "Content-Disposition";

    private static final String ACCESS_CONTROL_EXPOSE = "Access-Control-Expose-Headers";

    private static final String CHARACTER = "UTF-8";

    @Override
    public void dynamicImportExcel(MultipartFile file) throws IOException {
    
        String batchNumber = IdUtil.simpleUUID();
        EasyExcel.read(file.getInputStream(),
                        OrderRsp.class,
                        new OrderImportListener(importResultService, file.getOriginalFilename(), batchNumber))
                .sheet().headRowNumber(1).doRead();
    }

    @Override
    public void dynamicExportExcel(OrderExportParam param,
                                   HttpServletResponse response) throws IOException {
    
        long startTime = System.currentTimeMillis();

        LambdaQueryWrapper<ImportResultDetail> detailWrapper = new LambdaQueryWrapper<>();
        detailWrapper.eq(ImportResultDetail::getBatchNumber, param.getBatchNumber());

        List<ImportResultDetail> allDetailList = new ArrayList<>();
        int startIndex = 1;
        while (true) {
    
            int startParam = (startIndex - 1) * PAGE_SIZE;
            int pageIndex = (int) Math.ceil((double) startParam / (double) PAGE_SIZE + 1);
            Page<ImportResultDetail> pageQuery = new Page<>(pageIndex, PAGE_SIZE, false);
            Page<ImportResultDetail> detailByPage = detailMapper.selectPage(pageQuery, detailWrapper);

            List<ImportResultDetail> detailList = detailByPage.getRecords();
            if (CollectionUtils.isEmpty(detailList)) {
    
                break;
            }
            allDetailList.addAll(detailList);
            startIndex++;
        }

        ServletOutputStream outputStream = exportHeader(response, " Dynamically export information .xlsx");
        ExcelWriter excelWriter = EasyExcelFactory.write(outputStream)
                .registerWriteHandler(new CellWriteWeight()).build();

        // Query header 
        LambdaQueryWrapper<ImportResult> resultWrapper = new LambdaQueryWrapper<>();
        resultWrapper.eq(ImportResult::getBatchNumber, param.getBatchNumber());
        ImportResult importResult = resultMapper.selectOne(resultWrapper);

        List<ExcelHeader> excelHeaders = JSON.parseArray(importResult.getHeaders(), ExcelHeader.class);

        writeDate(excelWriter, allDetailList, excelHeaders);

        // Closed flow 
        excelWriter.finish();
        outputStream.flush();

        long endTime = System.currentTimeMillis();
        log.info(" Dynamic export takes time :{}", endTime - startTime);
    }

    // Write data 
    private void writeDate(ExcelWriter excelWriter, List<ImportResultDetail> detailList, List<ExcelHeader> headers) {
    
        // obtain sheet object 
        WriteSheet writeSheet = EasyExcel.writerSheet(" Import result details ").sheetNo(0)
                .head(EasyExcelUtils.headers(headers))
                .needHead(true)
                .registerWriteHandler(EasyExcelUtils.getStyleStrategy())
                .build();
        List<List<Object>> allList = new ArrayList<>();
        for (ImportResultDetail detail : detailList) {
    
            allList.addAll(EasyExcelUtils.dataList(headers, detail));
        }
        // towards sheet Write data 
        excelWriter.write(allList, writeSheet);
    }

    public ServletOutputStream exportHeader(HttpServletResponse response,
                                            String fileName) throws IOException {
    
        response.setContentType(CONTENT_TYPE);
        response.setHeader(ACCESS_CONTROL_EXPOSE, CONTENT_DISPOSITION); // So that the front end can get the file name, decode and convert it into Chinese 
        response.setHeader(CONTENT_DISPOSITION, "attachment; filename=" + URLEncoder.encode(fileName, CHARACTER));
        return response.getOutputStream();
    }

}

Dynamically import listeners

package cn.com.easyExcel.excel.listener;

import cn.com.easyExcel.excel.converter.HeadPropertiesConverter;
import cn.com.easyExcel.pojo.ExcelHeader;
import cn.com.easyExcel.pojo.ImportResult;
import cn.com.easyExcel.pojo.ImportResultDetail;
import cn.com.easyExcel.pojo.OrderRsp;
import cn.com.easyExcel.service.ImportResultService;
import cn.com.easyExcel.util.BeanCopyUtils;
import cn.hutool.core.util.StrUtil;
import com.alibaba.excel.context.AnalysisContext;
import com.alibaba.excel.metadata.data.ReadCellData;
import com.alibaba.excel.read.listener.ReadListener;
import com.alibaba.excel.util.ListUtils;
import com.alibaba.fastjson.JSON;
import lombok.extern.slf4j.Slf4j;

import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.atomic.AtomicInteger;

@Slf4j
public class OrderImportListener implements ReadListener<OrderRsp> {
    

    private final ImportResultService importResultService;

    private final String fileName;

    private final String batchNumber;

    /** *  every other 100 Storage database , Then clean up list, Convenient for memory recycling  */
    private static final int BATCH_COUNT = 100;

    // Total number of imports per batch 
    AtomicInteger total = new AtomicInteger(0);

    private String headers;

    // Cache data 
    private List<ImportResultDetail> cachedDataList = ListUtils.newArrayListWithExpectedSize(BATCH_COUNT);

    public OrderImportListener(ImportResultService importResultService,
                               String fileName,
                               String batchNumber) {
    
        this.importResultService = importResultService;
        this.fileName = fileName;
        this.batchNumber = batchNumber;
    }

    /** *  Get the attribute and name of the header  */
    public void invokeHead(Map<Integer, ReadCellData<?>> headMap, AnalysisContext context) {
    
        Map<String, String> propHead = new HashMap<>();
        Map<String, String> propMap = HeadPropertiesConverter.getHeadProperty(propHead, context.readWorkbookHolder().getClazz());

        List<ExcelHeader> headerList = new ArrayList<>();

        assert propMap != null;
        for (Map.Entry<Integer, ReadCellData<?>> headMapEntry : headMap.entrySet()) {
    
            log.info("head Of key:{},head Of value:{} ",headMapEntry.getKey(), headMapEntry.getValue());
            for (Map.Entry<String, String> propMapEntry : propMap.entrySet()) {
    
                if (StrUtil.equals(propMapEntry.getKey(), headMapEntry.getValue().getStringValue())) {
    
                    headerList.add(ExcelHeader.builder().name(propMapEntry.getKey()).prop(propMapEntry.getValue()).build());
                }
            }
        }
        headers = JSON.toJSONString(headerList);

        List<ExcelHeader> excelHeaders = JSON.parseArray(headers, ExcelHeader.class);
        log.info("excelHeaders:{}", excelHeaders);

    }

    @Override
    public void invoke(OrderRsp orderRsp, AnalysisContext analysisContext) {
    
        log.info(" Parse to a piece of data ：{}", JSON.toJSONString(orderRsp));
        total.addAndGet(1);

        ImportResultDetail importResultDetail = BeanCopyUtils.copyBean(orderRsp, ImportResultDetail::new);
        importResultDetail.setBatchNumber(batchNumber);
        cachedDataList.add(importResultDetail);

        if (cachedDataList.size() >= BATCH_COUNT) {
    
            batchSaveData();
            //  Storage complete cleaning  list
            cachedDataList = ListUtils.newArrayListWithExpectedSize(BATCH_COUNT);
        }
    }

    @Override
    public void doAfterAllAnalysed(AnalysisContext analysisContext) {
    
        // We also need to save data here , Make sure that the last legacy data is also stored in the database 
        batchSaveData();

        ImportResult importResult = ImportResult.builder()
                .batchNumber(batchNumber)
                .fileName(fileName)
                .uploadTime(new Date())
                .total(total.get())
                .headers(headers).build();
        importResultService.insertImportResult(importResult);
    }

    public void batchSaveData(){
    
        importResultService.batchInsertResultDetails(cachedDataList);
    }
}

HeadPropertiesConverter： Use reflection conversion excel Header

package cn.com.easyExcel.excel.converter;

import com.alibaba.excel.annotation.ExcelProperty;

import java.lang.reflect.Field;
import java.util.Map;

public class HeadPropertiesConverter {
    

    public static Map<String, String> getHeadProperty(Map<String, String> propHead, Class<?> clazz) {
    
        Field[] fields = clazz.getDeclaredFields();     // Mode two ：Field[] fields = ReflectUtil.getFields(clazz);

        if (fields.length == 0) {
    
            return null;
        }
        for (Field field : fields) {
    
            if (field.getAnnotation(ExcelProperty.class) != null) {
    
                propHead.put(field.getAnnotation(ExcelProperty.class).value()[0], field.getName());
            }
        }
        return propHead;
    }
}

Entity

Import results ： ImportResult.java

package cn.com.easyExcel.pojo;

import com.baomidou.mybatisplus.annotation.TableName;
import io.swagger.annotations.ApiModel;
import io.swagger.annotations.ApiModelProperty;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

import java.util.Date;

@Builder
@Data
@NoArgsConstructor
@AllArgsConstructor
@TableName("tb_import_result")
@ApiModel(value = "ImportResult", description = " Import results ")
public class ImportResult {
    

    @ApiModelProperty(value = " Batch number of each import (uuid)")
    private String batchNumber;

    @ApiModelProperty(value = " Upload file name ")
    private String fileName;

    @ApiModelProperty(value = " Upload time ")
    private Date uploadTime;

    @ApiModelProperty(value = " total ")
    private int total;

    @ApiModelProperty(value = "excel Header ", hidden = true)
    private String headers;

}

Import result details ：ImportResultDetail.java

package cn.com.easyExcel.pojo;

import com.baomidou.mybatisplus.annotation.IdType;
import com.baomidou.mybatisplus.annotation.TableId;
import com.baomidou.mybatisplus.annotation.TableName;
import io.swagger.annotations.ApiModel;
import io.swagger.annotations.ApiModelProperty;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

import java.time.LocalDateTime;

@Data
@NoArgsConstructor
@AllArgsConstructor
@TableName("tb_import_result_detail")
@ApiModel(value = "ImportResultDetail", description = " Import result details ")
public class ImportResultDetail {
    
    @ApiModelProperty(value = " The import result is obvious id", hidden = true)
    @TableId(value = "id", type = IdType.AUTO)
    private Long id;

    @ApiModelProperty(value = " Import batch number ")
    private String batchNumber;

    @ApiModelProperty(" Total sum , In minutes ")
    private Long totalPay;

    @ApiModelProperty(" Amount paid . Company : branch . Such as :20007, Express :200 element 7 branch ")
    private Long actualPay;

    @ApiModelProperty(" Payment type ,1、 Pay online ,2、 Cash on Delivery ")
    private Boolean paymentType;

    @ApiModelProperty(" Postage . Company : branch . Such as :20007, Express :200 element 7 branch ")
    private Long postFee;

    @ApiModelProperty(" Order creation time ")
    private LocalDateTime createTime;

    @ApiModelProperty(" Logistics name ")
    private String shippingName;

    @ApiModelProperty(" Logistics order No ")
    private String shippingCode;

    @ApiModelProperty(" Buyer message ")
    private String buyerMessage;

    @ApiModelProperty(" Buyer's nickname ")
    private String buyerNick;

    @ApiModelProperty(" Has the buyer commented on ,0 Not evaluated ,1 Evaluated ")
    private Boolean buyerRate;

    @ApiModelProperty(" Harvest address （ province ）")
    private String receiverState;

    @ApiModelProperty(" Harvest address （ City ）")
    private String receiverCity;

    @ApiModelProperty(" Harvest address （ District / county ）")
    private String receiverDistrict;

    @ApiModelProperty(" Harvest address （ The street 、 Address and other detailed addresses ）")
    private String receiverAddress;

    @ApiModelProperty(" Receiver's mobile phone ")
    private String receiverMobile;

    @ApiModelProperty(" Post code of consignee ")
    private String receiverZip;

    @ApiModelProperty(" Consignee ")
    private String receiver;

    @ApiModelProperty(" Invoice type (0 No invoice 1 Ordinary invoice ,2 Electronic invoice ,3 VAT invoice )")
    private Integer invoiceType;

    @ApiModelProperty(" Source of order ：1:app End ,2：pc End ,3：M End ,4： Wechat end ,5： mobile phone qq End ")
    private Integer sourceType;

}

Header ：ExcelHeader.java

package cn.com.easyExcel.pojo;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;

@Data
@Builder
@AllArgsConstructor
@NoArgsConstructor
public class ExcelHeader {
    

    private String name;

    private String prop;
}

Core code ideas

The key of dynamic import lies in the invokeHead Method , Using reflection to get the header Attribute field and Chinese name . Piece together name and prop Of JSON character , And the front table header The format is consistent .

The core method of dynamic export ：writeDate, It also uses reflection , according to header The content of , Take out ImportResultDetail Some fields of are displayed to the user .

Conclusion

Close test available . You can test by yourself , If you find it useful , Feel free to leave a comment .