Large scale background export excel cannot be concurrent

In recent days, , Complete a function . You need to export data in the database in the background , then , Generate Excel Send to front end .

The original program , yes C# Written , Adopt the principle of simplicity , Continue to realize on the original project .

The first problem is , The original ApiController I won't support it Response grammar . therefore , Change it to Controller Realization .

The second question is , yes Excel Export library for .

In the beginning , Try to use CSV As Excel export , Some data will have format problems , therefore , Finally adopted Excel Export Library . Last , After synthesis , choose NPOI library . The sample code is as follows ：

public class FileDownController : Controller
{
    private void createExport(List<AccountInfoAndFee> list, out string strRealPath)
	{
		HSSFWorkbook workbook = new HSSFWorkbook();
		FileStream fileNum = null;
		
		try
		{
			// 1. Check whether the root folder exists , If it doesn't exist, create a folder 
			if (!Directory.Exists(" File path to save "))
			{
				Directory.CreateDirectory(CommonFiles.FILE_ROOT_PATH + id);
			}

			strRealPath =  The actual file path ; // You need to change 

			//  workbook  
			ISheet sheet = workbook.CreateSheet(" data ");

			//  Worksheet  
			IRow row = sheet.CreateRow(0);
			createExcleHeader(ref row, ref sheet);

			int j = 0;

			int k = 0;

			//while(k++ < 10)
			//{
			for (int i = 0; i < list.Count(); i++)
			{
				// Data processing 
				IRow rowTmp = sheet.CreateRow(i + 1);
				rowTmp.CreateCell(0).SetCellValue(""); // Subsequent data are also processed 
			}

			// 4. Generate the file  
			
			fileNum = new FileStream(strRealPath + fileName + ".xls", FileMode.Create);
			workbook.Write(fileNum);
			fileNum.Flush();
			fileNum.Close();
			workbook.Clear();
			workbook.Close();

			Log.Error(" file " + strRealPath + fileName + "  Total number of lines : " + list.Count().ToString());
		}
		catch (Exception ex)
		{
			Log.Error(ex, ex.StackTrace);
			Log.Error(ex, ex.Message);
			throw ex;
		}
		finally
		{
			workbook.Close();
			if (fileNum != null)
			{
				fileNum.Close();
			}
		}
	}

    private void getFiles(string fileSrcName)
     {
	    Response.AppendHeader("Content-Disposition", "attachment;filename=" + filename);
	    Response.ContentType = "application/ms-excel";
	    Response.TransmitFile(fileSrcName);
	    Response.Flush();
	    Response.End();

	    removeFiles(strFilePath);
    }
}

In the use of NPOI when , One problem found is ：HSSFWorkbook Only support 65535 That's ok . The first thing to consider is , Replace with XSSFWorkbook. But found , Even if it's XSSFWorkbook, There is still a maximum number of files . So the solution is ： Also use HSSFWorkbook, But to 65535 Line time , Just generate a file .

But this problem comes again , Just HTTP Transmitted as 1 File . Method is ： Compress Excel File for 1 Compressed files .

But there are still important problems ： That's it ,NPOI When writing large files , Only one write process is supported , Otherwise it will go wrong .

Suddenly thought of 《Hadoop principle 》 What I said ： It was found that , Even the largest elephant , Its pulling ability is also limited ; But people will not cultivate bigger elephants , Instead, let several elephants pull the cart .

therefore , Methods are provided ： Provide multiple servers , Serve multiple requests separately .

therefore , Problem solving .