Divide and conquer, upload large files in pieces

Here we involve an algorithm —— Branching algorithm

In the recent advertising platform project , Because customers need to provide self certification videos , The certificate is the authenticity of my company .

Today, , The resolution of video is getting higher , Often accompanied by the huge video memory , A few minutes of video needs several g Of memory , It's definitely not possible to upload using ordinary methods , Large files are slow , Impact on customer experience , To solve this problem , I used the method of uploading in pieces , Will a big file , Divide into n Upload a small file , Speed up transmission efficiency ！

Here we involve an algorithm —— Branching algorithm

The concept of fragment upload is the divide and conquer algorithm , Divide and rule , Put a big question , Turn into several small problems , Solve ！！

Front end segmentation

First of all, we do it at the front , Fragment transmission to the back end , So here I'm going to use theta vue3,ui The framework is ant-desgin, The controls used are a-upload, Users click upload to select the file to upload .

Then we can get the name of the file , And the size , Then define the size you want to slice , Here you can write a judgment , It depends on the size of the file you want to upload , Define the size of your slice , If the file is too small , There is no need to slice , Occupancy resources . At this time, we can get the total number of films , Because there will be remainder , So what I use is Math.ceil Rounding up , Get the total number of films . At this time, we can get the start position and end position of each slice , And then through slice Slice the file , Send it to the backend for processing ！

// Upload method 
    handleChange:function(file){

      // Get file size     1024 byte    1kb
      console.log(file.file)
      
      var size = file.file.size
      this.size = size
      // this.filename = file.file.name

      // Define fragment size 
      var shardSize = 1024 * 100

      // The total number of films 
      // Rounding up 
      this.shardCount = Math.ceil(size / shardSize)

      // Slicing operation 
      for (var i=0;i<this.shardCount; ++i){
        // Starting position 
        var start = i * shardSize

        // End position 
        var end = Math.min(size,start+shardSize)

        // section 
        var shardfile = file.file.slice(start,end)

        this.pushshard(shardfile,file.file.name,i)
      }
      


    },

So far, the front-end slicing logic is completed ！

---

Back end asynchronous write

Back end I use python Medium tronado The framework writes asynchronously , To avoid blocking caused by synchronous writing , install aiofiles The library cooperates tornado Asynchronous mechanism of , It can improve the efficiency of writing

Here, the back end gets the file entity , file name , Sign a , First write the file entity asynchronously to the specified path , And then read it , Merge in order , I won't talk too much about file operation , They are all fixed grammars , There's no logic , The following code shows , Please watch for yourself

#  Patch uploading 
class SliceUploadHandler(BaseHandler):
    # Merging and slicing 
    async def put(self):
        filename = self.get_argument("filename",None)
        count = 0
        size = self.get_argument("size",None)

        try:
            filesize = os.path.getsize("./static/upload/{}".format(filename))
        except Exception as e:
            print(str(e))
            filesize = 0

        if int(size) != filesize:

            # Open file handle asynchronously 
            async with aiofiles.open("./static/upload/{}".format(filename),"ab") as file:
                while True:
                    try:
                        # Read slices 
                        shard_file = open("./static/upload/{}_{}".format(filename,count),"rb")
                        # Asynchronous write 
                        await file.write(shard_file.read())
                        # Close the handle manually 
                        shard_file.close()

                    except Exception as e:

                        print(str(e))

                        break

                    count = count + 1

                self.finish({"errcode":0,"msg":" The merger is over "})


    #  Fragment file receiving 
    async def post(self):
        # Receive fragment entities 
        file = self.request.files["file"][0]

        # Get subscript 
        count = self.get_argument("count",None)

        # Get the file name 
        filename = self.get_argument("filename",None)


        # Get file content 
        content = file["body"]

        # Asynchronous write 
        async with aiofiles.open("./static/upload/{}_{}".format(filename,count),"wb") as file:

            # asynchronous 
            await file.write(content)

        self.finish({"errcode":0,"msg":" Slice upload succeeded "})

So far, the debugging of the front and rear ends is completed , besides , In a real super large file transfer scenario , Due to network or other factors , It is likely to cause the fragmentation task to be interrupted , At this point, you need to respond quickly through degradation , Back to the bottom data , Avoid users' long waiting , Here we use a method based on Tornado Of Apscheduler Library to schedule fragment tasks .