How to use multithreading to export excel under massive data? Source code attached!

Click on the above “Java selected ”, choice “ Set to star ”

Don't ask people why , Ask yourself why ！

below Have a pleasant surprise A message will be back , Have a craigslist ！

Make progress every day , It's the beginning of success ...

Preface

The company's project recently has a need ： Report export . The whole system down , At least more than 100 reports need to be exported . How to export the report gracefully at this time , Releasing productivity is very important . The following is to share the usage and implementation ideas of this tool class .

Function point of realization

For each report, the same operation , We naturally pull out , This is very simple . And most importantly ： How to well encapsulate the different operations of each report , Improve reusability as much as possible ; For the above principles , It mainly realizes the following key function points ：

• Export any type of data

• Set the header freely

• Freely set the export format of the field

Using examples

As mentioned above, this tool class implements three function points , Naturally, you can set these three points when using ：

• Set data list

• Set the header

• Set field format

Below export Function can directly return a to the client excel data , among productInfoPos List the data to be exported ,ExcelHeaderInfo Used to save header information , Include header name , First column of header , Tail column , First line , Biko .

Because the default exported data format is string , So we need another Map Parameter is used to specify the format type of a field （ For example, number type , Decimal type 、 The date type ）. Here you know how to use it , These parameters will be explained in detail below .

@Override
    public void export(HttpServletResponse response, String fileName) {
        //  Data to be exported 
        List<TtlProductInfoPo> productInfoPos = this.multiThreadListProduct();
        ExcelUtils excelUtils = new ExcelUtils(productInfoPos, getHeaderInfo(), getFormatInfo());
        excelUtils.sendHttpResponse(response, fileName, excelUtils.getWorkbook());
    }

    //  Get header information 
    private List<ExcelHeaderInfo> getHeaderInfo() {
        return Arrays.asList(
                new ExcelHeaderInfo(1, 1, 0, 0, "id"),
                new ExcelHeaderInfo(1, 1, 1, 1, " Name of commodity "),

                new ExcelHeaderInfo(0, 0, 2, 3, " classification "),
                new ExcelHeaderInfo(1, 1, 2, 2, " type ID"),
                new ExcelHeaderInfo(1, 1, 3, 3, " Category name "),

                new ExcelHeaderInfo(0, 0, 4, 5, " brand "),
                new ExcelHeaderInfo(1, 1, 4, 4, " brand ID"),
                new ExcelHeaderInfo(1, 1, 5, 5, " The brand name "),

                new ExcelHeaderInfo(0, 0, 6, 7, " The store "),
                new ExcelHeaderInfo(1, 1, 6, 6, " The store ID"),
                new ExcelHeaderInfo(1, 1, 7, 7, " Shop name "),

                new ExcelHeaderInfo(1, 1, 8, 8, " Price "),
                new ExcelHeaderInfo(1, 1, 9, 9, " stock "),
                new ExcelHeaderInfo(1, 1, 10, 10, " sales "),
                new ExcelHeaderInfo(1, 1, 11, 11, " Insert time "),
                new ExcelHeaderInfo(1, 1, 12, 12, " Update time "),
                new ExcelHeaderInfo(1, 1, 13, 13, " Whether the record has been deleted ")
        );
    }

    //  Get formatting information 
    private Map<String, ExcelFormat> getFormatInfo() {
        Map<String, ExcelFormat> format = new HashMap<>();
        format.put("id", ExcelFormat.FORMAT_INTEGER);
        format.put("categoryId", ExcelFormat.FORMAT_INTEGER);
        format.put("branchId", ExcelFormat.FORMAT_INTEGER);
        format.put("shopId", ExcelFormat.FORMAT_INTEGER);
        format.put("price", ExcelFormat.FORMAT_DOUBLE);
        format.put("stock", ExcelFormat.FORMAT_INTEGER);
        format.put("salesNum", ExcelFormat.FORMAT_INTEGER);
        format.put("isDel", ExcelFormat.FORMAT_INTEGER);
        return format;
    }

Realization effect

Source code analysis

ha-ha , Analyze your own code , I little interesting . Because it's inconvenient to post too much code , You can come first github On clone Source code , Come back and read the article .

Source code address

https://github.com/dearKundy/excel-utils

LZ The use of poi 4.0.1 Version of this tool , If you want to export massive data, you have to use it naturally SXSSFWorkbook This component . About poi I won't say more about the specific usage of , Here is mainly to explain how to poi For encapsulation use .

Member variables

Let's focus on ExcelUtils This class , This class is the core of the export implementation , Let's first look at three member variables .

private List list;
    private List<ExcelHeaderInfo> excelHeaderInfos;
    private Map<String, ExcelFormat> formatInfo;

list

This member variable is used to save the data to be exported .

ExcelHeaderInfo

This member variable is mainly used to save header information , Because we need to define multiple header information , So you need to use a list to save ,ExcelHeaderInfo The constructor is as follows

ExcelHeaderInfo(int firstRow, int lastRow, int firstCol, int lastCol, String title)

• firstRow： The first row of the position occupied by the header

• lastRow： The last line of the position occupied by the header

• firstCol： The first column of the position occupied by the header

• lastCol： The last line of the position occupied by the header

• title： The name of the header

ExcelFormat

This parameter is mainly used to format fields , We need to agree in advance to convert to that format , It can't be decided by the user . So we define a variable of enumeration type , This enumeration class has only one member variable of string type , Used to save the format you want to convert , for example FORMAT_INTEGER Is to convert to an integer . Because we need to accept the conversion format of multiple fields , So it defines a Map Type to receive , This parameter can be omitted （ The default format is string ）.

public enum ExcelFormat {

    FORMAT_INTEGER("INTEGER"),
    FORMAT_DOUBLE("DOUBLE"),
    FORMAT_PERCENT("PERCENT"),
    FORMAT_DATE("DATE");

    private String value;

    ExcelFormat(String value) {
        this.value = value;
    }

    public String getValue() {
        return value;
    }
}

The core approach

1. Create header

This method is used to initialize the header , The key to creating a header is poi in Sheet Class addMergedRegion(CellRangeAddress var1) Method , This method is used for cell fusion .

We'll traverse ExcelHeaderInfo list , According to each ExcelHeaderInfo Coordinate information for cell fusion , Then create cells in the first row and the first column of each cell after fusion , Then assign a value to the cell , Through the above steps, you can complete any type of header setting .

//  Create header 
    private void createHeader(Sheet sheet, CellStyle style) {
        for (ExcelHeaderInfo excelHeaderInfo : excelHeaderInfos) {
            Integer lastRow = excelHeaderInfo.getLastRow();
            Integer firstRow = excelHeaderInfo.getFirstRow();
            Integer lastCol = excelHeaderInfo.getLastCol();
            Integer firstCol = excelHeaderInfo.getFirstCol();

            //  The row spacing or column spacing is greater than 0 Before cell fusion 
            if ((lastRow - firstRow) != 0 || (lastCol - firstCol) != 0) {
                sheet.addMergedRegion(new CellRangeAddress(firstRow, lastRow, firstCol, lastCol));
            }
            //  Get the first row position of the current header 
            Row row = sheet.getRow(firstRow);
            //  Create a new cell in the first row and column of the header 
            Cell cell = row.createCell(firstCol);
            //  Assignment cell 
            cell.setCellValue(excelHeaderInfo.getTitle());
            cell.setCellStyle(style);
            sheet.setColumnWidth(firstCol, sheet.getColumnWidth(firstCol) * 17 / 12);
        }
    }

2. Conversion data

Before text assignment , We first need to convert the original data list into a two-dimensional array of strings , The reason why it is converted to string format is that it can deal with various types uniformly , Then we can switch back if necessary .

//  Convert the original data into a two-dimensional array 
    private String[][] transformData() {
        int dataSize = this.list.size();
        String[][] datas = new String[dataSize][];
        //  Get the number of columns in the report 
        Field[] fields = list.get(0).getClass().getDeclaredFields();
        //  Get the field name array of the entity class 
        List<String> columnNames = this.getBeanProperty(fields);
        for (int i = 0; i < dataSize; i++) {
            datas[i] = new String[fields.length];
            for (int j = 0; j < fields.length; j++) {
                try {
                    //  assignment 
                    datas[i][j] = BeanUtils.getProperty(list.get(i), columnNames.get(j));
                } catch (Exception e) {
                    LOGGER.error(" Failed to get object property value , official account ：Java selected , Have a pleasant surprise ");
                    e.printStackTrace();
                }
            }
        }
        return datas;
    }

In this method, we use reflection technology , It cleverly realizes the export of any type of data （ Any type here refers to any report type , Different reports , The exported data must be different , So in Java The entity classes in the implementation are certainly different ）. If you want to List Convert to the corresponding two-dimensional array , We need to know the following information :

• The number of columns in a two-dimensional array

• The number of rows in a two-dimensional array

• The value of each element of a two-dimensional array

What if you get the above three information ？

Through... In reflection Field[] getDeclaredFields() This method gets all the fields of the entity class , So as to indirectly know how many columns there are

List The size of is just the number of rows of the two-dimensional array

Although the field names of each entity class are different , So we really can't get the value of a field of the entity class ？ No, it isn't , You need to know , You have reflection , You have the world , What else can't you do . We don't use reflection directly here , It's using a name called BeanUtils Tools for , This tool can easily help us to assign fields and obtain field values for an entity class .

It's simple , adopt BeanUtils.getProperty(list.get(i), columnNames.get(j)) This line of code , We get the entity list.get(i) In the name of columnNames.get(j) The value of this field .list.get(i) Of course, we traverse the entity class of the original data , and columnNames The list is an array of all field names of an entity class , It is also obtained through reflection , Specific implementation can refer to LZ Source code .

3. Assignment text

The text here specifies the formal table data content , In fact, there are not many strange sexual skills , The main functions have been realized above , Here is mainly the processing of cell assignment and export format （ Mainly to export excel Convenient calculation can be carried out after ）.

//  Create text 
    private void createContent(Row row, CellStyle style, String[][] content, int i, Field[] fields) {
        List<String> columnNames = getBeanProperty(fields);
        for (int j = 0; j < columnNames.size(); j++) {
            if (formatInfo == null) {
                row.createCell(j).setCellValue(content[i][j]);
                continue;
            }
            if (formatInfo.containsKey(columnNames.get(j))) {
                switch (formatInfo.get(columnNames.get(j)).getValue()) {
                    case "DOUBLE":
                        row.createCell(j).setCellValue(Double.parseDouble(content[i][j]));
                        break;
                    case "INTEGER":
                        row.createCell(j).setCellValue(Integer.parseInt(content[i][j]));
                        break;
                    case "PERCENT":
                        style.setDataFormat(HSSFDataFormat.getBuiltinFormat("0.00%"));
                        Cell cell = row.createCell(j);
                        cell.setCellStyle(style);
                        cell.setCellValue(Double.parseDouble(content[i][j]));
                        break;
                    case "DATE":
                        row.createCell(j).setCellValue(this.parseDate(content[i][j]));
                }
            } else {
                row.createCell(j).setCellValue(content[i][j]);
            }
        }
    }

The core methods of exporting tool classes are almost finished , Let's talk about multi-threaded query .

Two more points

1. Multi thread query data

The ideal is full , The reality is still a little bony .LZ Although the 50w The data is created separately 20 A thread to query , But the overall efficiency is not 50w/20, But just a few seconds faster , If you know the reason, you can leave me a message and discuss it together .

Let's talk about the specific ideas first ： Because multiple threads execute simultaneously , You can't guarantee which thread will finish executing first , But we have to ensure the consistency of data order . Here we use Callable Interface , By implementing Callable The thread of the interface can have the return value , We get the query results of all sub threads , Then merge into a result set . So how to ensure the order of consolidation ?

Let's create a FutureTask Type of List, The FutureTask The type of is the result set returned .

List<FutureTask<List<TtlProductInfoPo>>> tasks = new ArrayList<>();

Every time we start a thread , The thread's FutureTask Add to tasks In the list , such tasks The order of elements in the list is the order in which we start the thread .

FutureTask<List<TtlProductInfoPo>> task = new FutureTask<>(new listThread(map));
log.info(" Start to query the number {} Bar start {} Bar record ", i * THREAD_MAX_ROW, THREAD_MAX_ROW);
new Thread(task).start();
//  Add tasks to tasks In the list 
tasks.add(task);

Next , That's the sequential plug value , We start from... In order tasks Take out... From the list FutureTask, And then execute FutureTask Of get() Method , This method blocks the thread that calls it , Know the return result . Such a set of cycles , This completes the sequential storage of all data .

for (FutureTask<List<TtlProductInfoPo>> task : tasks) {
        try {
            productInfoPos.addAll(task.get());
        } catch (Exception e) {
            e.printStackTrace();
        }
}

2. How to solve interface timeout

If you need to export massive data , There may be a problem ： Interface timeout , The main reason is that the whole export process takes too long . In fact, it's also very easy to solve , The response time of the interface is too long , We can shorten the response time . We use asynchronous programming solutions , There are many ways to implement asynchronous programming , Here we use the simplest spring Medium Async annotation , The method with this annotation can immediately return the response result .

About how annotations are used , You can check it yourself , Let's talk about the key implementation steps ：

1、 Write asynchronous interface , This interface is responsible for receiving the export request from the client , Then start exporting （ Be careful ： The export here does not return directly to the client , Instead, download to the local server ）, As long as the export instruction is issued , You can immediately return this to the client excel The unique mark of the file （ Used to find the file later ）, End of interface .

2、 To write excel State interface , The client gets excel After the unique flag of the file , Start polling every second and call the interface to check excel Export status of the file

3、 Write a local return from the server excel File interface , If the client checks excel Successfully downloaded to the local server , At this time, you can request the interface to download files directly .

In this way, the problem of interface timeout can be solved .

Source code address

https://github.com/dearKundy/excel-utils

Source code taking posture

1、 Build table （ Insert the data by yourself ）

CREATE TABLE `ttl_product_info` (
  `id` int(11) NOT NULL AUTO_INCREMENT COMMENT ' Record unique identification ',
  `product_name` varchar(50) NOT NULL COMMENT ' Name of commodity ',
  `category_id` bigint(20) NOT NULL DEFAULT '0' COMMENT ' type ID',
  `category_name` varchar(50) NOT NULL COMMENT ' Redundant classification name - Avoid cross table join',
  `branch_id` bigint(20) NOT NULL COMMENT ' brand ID',
  `branch_name` varchar(50) NOT NULL COMMENT ' Redundant brand name - Avoid cross table join',
  `shop_id` bigint(20) NOT NULL COMMENT ' goods ID',
  `shop_name` varchar(50) NOT NULL COMMENT ' Redundant store name - Avoid cross table join',
  `price` decimal(10,2) NOT NULL COMMENT ' The current price of the goods - Belongs to hot data , And price changes need to be recorded , Need price details ',
  `stock` int(11) NOT NULL COMMENT ' stock - Hot data ',
  `sales_num` int(11) NOT NULL COMMENT ' sales ',
  `create_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT ' Insert time ',
  `update_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT ' Update time ',
  `is_del` tinyint(3) unsigned NOT NULL DEFAULT '0' COMMENT ' Whether the record has been deleted ',
  PRIMARY KEY (`id`),
  KEY `idx_shop_category_salesnum` (`shop_id`,`category_id`,`sales_num`),
  KEY `idx_category_branch_price` (`category_id`,`branch_id`,`price`),
  KEY `idx_productname` (`product_name`)
) ENGINE=InnoDB AUTO_INCREMENT=15000001 DEFAULT CHARSET=utf8 COMMENT=' Commodity information table ';

2、 Run the program

3、 Enter... In the address bar of the browser ：http://localhost:8080/api/excelUtils/export Download it

author ： You are at my door

https://juejin.cn/post/6844903779167535117

official account “Java selected ” The published content indicates the source of , All rights reserved （ Those whose copyright cannot be verified or whose source is not indicated all come from the Internet , Reprinted , The purpose of reprinting is to convey more information , The copyright belongs to the original author . If there is any infringement , Please contact the , The author will delete the first time ！

------ THE END ------

Excellent materials , Great benefits ！

>Java Interview questions <
3000+ Avenue BAT Big factory interview questions online brush , newest 、 The most complete Java Interview questions ！

* Java Advanced learning materials
* Java self-taught 、 Advanced roadmap free of charge

Hope to select    Click on the title to jump to

Mysql Indexes ： illustrated , Delve into the principle and use of indexing ！

Byte intern delete GB All of the following models ！ Net friend ：“ rendering outstanding service ” 了 ！

Linus：“ I deleted Linux, Because it's rubbish ！”

It's not me , This kind of IDEA You really haven't used the plug-in yet ！

IT The literary world is surprised to see literary giants ！ Huawei leaders and Alibaba P10 Make complaints about ！

Oh my god ！ Implement... In code 3D Ice mound , It's too romantic ！

What's the most garbage code you've ever seen ？19 A piece of junk code ！

In tencent , How to do Code Review？

  technology Communication group ！

Many people have asked recently , Is there any readers Communication group ！ Want to know how to join ？ It's very simple , Friends with similar interests , Just click on the card below , reply “ Add group ”, that will do No routine Join the communication group ！

If the article helps , Looking at , Forward! ！