当前位置：网站首页>Large attachment fragment upload and breakpoint continuation
Large attachment fragment upload and breakpoint continuation
2022-06-13 03:05:00 【good_ boys】
Related implementation ：
https://segmentfault.com/a/1190000021945539?utm_source=tag-newest
https://mp.weixin.qq.com/s/f2Una_4u6xecEyrto47_Fg
Fragment uploading requires the cooperation of the front end , Split the accessories on the end , Serial or parallel upload fragmentation .
front end ：
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>JS Patch uploading - Fast upload </title>
</head>
<body>
<input type="file" name="slice" id="slice" >
<br/>
</body>
<script src="http://libs.baidu.com/jquery/1.8.3/jquery.min.js"></script>
<script type="text/javascript"> $("#slice").change(function(event) {
var file = $("#slice")[0].files[0]; PostFile(file,0); }); // Execute fragment upload function PostFile(file,i, uuid){
var name = file.name, // file name size = file.size, // Total size shardSize = 1 * 1024 * 1024, // With 1MB For a piece , The size of each slice shardSize = 1 * 1024 * 1024 shardCount = Math.ceil(size / shardSize); // The total number of films if(i >= shardCount){
console.log("done"); return; } //console.log(size,i+1,shardSize); // Total file size , for the first time , Slice size // var start = i * shardSize; var end = start + shardSize; var packet = file.slice(start, end); // Slice the file /* structure form Form to submit */ var form = new FormData(); form.append("fileMd5", fileMd5());// For the whole file md5, Get key string （ To be modified ） form.append("data", packet); //slice Method is used to cut out part of the file form.append("fileName", name); form.append("chunkSize", shardSize); form.append("chunkTotal", size); // The total number of films form.append("chunkIdx", i); // What is the current film $.ajax({
url: "http://localhost:8082/member/file/upload", type: "POST", data: form, //timeout:"10000", // Overtime 10 second async: true, // asynchronous dataType:"json", processData: false, // Very important , tell jquery Don't take form To deal with contentType: false, // Very important , Designated as false To form the right Content-Type success: function (msg) {
console.log(msg); // i++; // PostFile(file, i, uuid); /* Indicates that the last file was uploaded successfully , Go on for the next time */ if (msg.status == 201) {
form = ''; i++; PostFile(file, i, uuid); } else if (msg.status == 502) {
form = ''; /* After failure , Every time 2 Second, continue to transfer the file once */ setInterval(function () {
PostFile(file, i, uuid) }, 2000); } else if (msg.status == 200) {
// merge(uuid, name) console.log(" Upload successful "); } else if (msg.status == 500) {
console.log(' The first '+msg.i+' Time , The upload file is wrong ！'); } else {
console.log(' Unknown error '); } } }) } function fileMd5() {
return 'aaabbbccc'; // return 'xxxxxxxxxxxx4xxxyxxxxxxxxxxxxxxx'.replace(/[xy]/g, function (c) {
// var r = Math.random() * 16 | 0, // v = c == 'x' ? r : (r & 0x3 | 0x8); // return v.toString(16); // }); } </script>
</html>
Server side ：
// An highlighted block
public interface FileUploadService {
/** * Upload * @param uploadRequestDTO * @return * @throws IOException */
Boolean upload(FileUploadRequestDTO uploadRequestDTO) throws IOException;
FileUploadProcessResp getProcessByMd5(String fileMd5);
}
// An highlighted block
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.FileUtils;
import org.apache.logging.log4j.util.Strings;
import org.springframework.stereotype.Service;
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.util.*;
@Service
@Slf4j
public class FileUploadServiceImpl implements FileUploadService {
private static final String TMP_DIR_PATH = "/Users/didi/demo_data/testFile";
/** * Upload attachments in pieces * @param param * @return * @throws IOException */
@Override
public Boolean upload(FileUploadRequestDTO param) throws IOException {
initParam(param);
RandomAccessFile accessTmpFile = null;
try {
File tmpFile = createTmpFile(param);
accessTmpFile = new RandomAccessFile(tmpFile, "rw");
accessTmpFile.setLength(param.getChunkTotal());
// Locate the offset to the slice
accessTmpFile.seek(param.getOffset());
// Write the slice data
accessTmpFile.write(param.getData());
checkAndSetUploadProgress(param);
return true;
} catch (Exception e) {
log.error(e.getMessage(), e);
} finally {
accessTmpFile.close();
}
return false;
}
/** * Query the upload progress * @param fileMd5 * @return */
@Override
public FileUploadProcessResp getProcessByMd5(String fileMd5) {
String confFilePath = findConfFile(fileMd5);
File confFile = new File(confFilePath);
List<Integer> chunkStatus = new ArrayList<>();
int isComplete = 1;
try {
byte[] completeList = FileUtils.readFileToByteArray(confFile);
for (int i = 0; i < completeList.length ; i++) {
if (Byte.MAX_VALUE == completeList[i]){
chunkStatus.add(127);
} else {
chunkStatus.add(0);
isComplete = 0;
}
}
}catch (IOException e){
log.error(e.getMessage());
}
return FileUploadProcessResp.builder()
.chunkStatus(chunkStatus)
.isComplete(isComplete)
.build();
}
private String findConfFile(String fileMd5) {
String path = TMP_DIR_PATH + File.separator + fileMd5;
File dir = new File(path);
if (!dir.exists()){
return Strings.EMPTY;
}
String[] arr = dir.list();
List<String> strings = Arrays.asList(arr);
return strings.stream()
.filter(fileName -> fileName.substring(fileName.lastIndexOf(".")).equals("conf"))
.findFirst()
.orElseThrow(() -> new BizException(BizErrorCodeEnum.DATA_NO_FOUND));
}
/** * Initialize variable * @param param */
private void initParam(FileUploadRequestDTO param) {
param.setTmpDirPath(TMP_DIR_PATH + File.separator + param.getFileMd5());
param.setOffset(param.getChunkSize() * param.getChunkIdx());
param.setTmpFileName(param.getFileName() + "_tmp");
}
protected File createTmpFile(FileUploadRequestDTO param) {
String uploadDirPath = param.getTmpDirPath();
File tmpDir = new File(uploadDirPath);
if (!tmpDir.exists()) {
tmpDir.mkdirs();
}
return new File(uploadDirPath, param.getTmpFileName());
}
/** * Check and modify file upload progress , Uploaded marked as 127 */
public void checkAndSetUploadProgress(FileUploadRequestDTO param) throws IOException {
String fileName = param.getFileName();
File confFile = new File(param.getTmpDirPath(), fileName + ".conf");
RandomAccessFile accessConfFile = null;
try {
accessConfFile = new RandomAccessFile(confFile, "rw");
log.info("set part " + param.getChunkIdx() + " complete");
accessConfFile.setLength(param.getChunkTotal());
accessConfFile.seek(param.getChunkIdx());
accessConfFile.write(Byte.MAX_VALUE);
// Check that it's all done , If all of the arrays are 127( All the pieces have been uploaded successfully )
byte[] completeList = FileUtils.readFileToByteArray(confFile);
byte isComplete = Byte.MAX_VALUE;
for (int i = 0; i < completeList.length && isComplete == Byte.MAX_VALUE; i++) {
// And operation , If some parts are not completed, then isComplete No Byte.MAX_VALUE
isComplete = (byte) (isComplete & completeList[i]);
log.info("check part " + i + " complete?:" + completeList[i]);
}
// Uploading on slice is completed , Move attachment to final position
moveFile(param);
} catch (Exception e) {
log.error(e.getMessage(), e);
} finally {
accessConfFile.close();
}
}
private Boolean moveFile(FileUploadRequestDTO param) {
File oldFile = new File(param.getTmpDirPath(), param.getTmpFileName()); // Merged temporary large file
File newFile = new File(param.getTmpDirPath(), param.getFileName()); // New big file
FileUtils.moveFile(oldFile, newFile);
}
}
Entity object
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class FileUploadRequestDTO {
private String fileName;
private byte[] data;
private Integer chunkIdx;
private Integer chunkSize;
private Integer chunkTotal;
private String fileMd5;
/** * Custom variable */
private String tmpDirPath;
private Integer offset;
private String tmpFileName;
}
import lombok.Builder;
import lombok.Data;
import java.util.List;
/** * **/
@Data
@Builder
public class FileUploadProcessResp {
/** * Is it all done 0 Hang in the air 1 Completed */
private Integer isComplete;
/** * key It's the segment number ,value It is the status of partition completion ：127 Completed 0 Hang in the air */
private List<Integer> chunkStatus;
}
边栏推荐
- Rounding in JS
- Introduction to Kestrel_ Introduction to kestrel web server
- Stack: daily temperature
- Detailed explanation of data processing in machine learning (I) -- missing value processing (complete code attached)
- 二叉樹初始化代碼
- Retrofit easy to use
- Summary of the latest IOS interview questions in June 2020 (answers)
- Collection of IOS development interview and underlying learning videos
- JVM JMM (VI)
- Implementing fillet in custom dialog
猜你喜欢
Data processing in detailed machine learning (II) -- Feature Normalization
CDN single page reference of indexbar index column in vant framework cannot be displayed normally
MySQL transactions and locks (V)
專業的數據庫管理軟件：Valentina Studio Pro for Mac
Applet image component long press to identify supported codes
Introduction to facial expression recognition system - Technical Paper Edition
![[deep learning] fast Reid tutorial](/img/aa/8215be94109eefa0324430657e4098.jpg)
[deep learning] fast Reid tutorial
Review notes of RS data communication foundation STP
Introduction to redis (using redis, common commands, persistence methods, and cluster operations)
MySQL 8.0 installation free configuration method
随机推荐
JVM class loader (2)
Ijkplayer source code - rendering
Five old code farmers, program life review: peace of mind is not the place to go
Vscode liveserver use_ Liveserver startup debugging
Traverse the array and delete an element until it is deleted
OneNote User Guide (1)
Mp4 playback
Few-shot Unsupervised Domain Adaptation with Image-to-Class Sparse Similarity Encoding
Ijkplayer source code -- mnatemediaplayer of ijkmediaplayer
Simple use of leaflet - offline map scheme
[data analysis and visualization] key points of data drawing 12- importance of chart notes
vant实现移动端的适配
MySQL transactions and locks (V)
Pycharm installation pyqt5 and its tools (QT designer, pyuic, pyrcc) detailed tutorial
开源-校园论坛和资源共享小程序
Linked list: reverse linked list
HEAP[xxx.exe]: Invalid address specified to RtlValidateHeap( 0xxxxxx, 0x000xx)
JS deconstruction assignment
Professional database management software: Valentina Studio Pro for Mac
Linked list: the entry node of the link in the linked list