One of the general document service practice series

File service is a very basic function , Generally speaking , Each system will have similar upload avatars 、 Post pictures 、 Upload office files and other similar pictures or file upload functions .

The functions of this part of each project are actually similar , It's different , That is, the types of uploaded files are different , The size limits are different , The strength of access permission control is different , If there are many different projects in a company , It is very necessary to design a general file service to meet all similar file upload functions .

In design document service , There are several key attributes to consider ：

• The sensitivity of the file content

• Average file size

• How files are used 、 Frequency of use

• File access control

• Importance of documents

The file attributes listed above are considered from the file itself and use , Different attributes determine that different ways must be taken to treat ：

The sensitivity of the file content

What is the sensitivity of content ？ In the vernacular, it is the uploaded file , When the uploader is anonymous , Will the content of the file expose the uploader or others / thing / Sensitive information of the company . for example , In a system like finance or payment , The file contains the bank card number 、 Phone number 、 User ID number , Or the contract of the enterprise , Files containing such information can be said to be sensitive . On the contrary, it's like a picture that users casually send to a blog , It doesn't matter if you are inadvertently seen , This is not sensitive .

When we design document service , If you need to store sensitive files , Then we should fully consider the confidentiality of the contents of the document . Common measures that can be taken are ：

• Block storage ： Cut the file into many blocks . Put it in a different place , Even if you get a piece, you can't restore the contents of the file .

• Encrypted storage ： Before the file is written to disk and saved , Encrypt File Bytes . If Edison knew something about safety in those days , Then the audience will lose a big melon .

• Confusing storage ： When saving files , Rearrange the contents of the file with certain rules , The person who steals or sees it will not understand , So as to protect the confidentiality of documents

Average file size

The volume of files determines the efficiency of file access and storage . Is the reading speed of small files relative to large files 、 Save the write speed and other aspects should be fast ？ It's not . 

In Taobao system , Due to product pictures 、 Decoration pictures 、 Description pictures belong to small pictures , There are a large number of small files in the system , Or in machine learning , Similarly, a large number of small files will be used . For these massive small files , If conventional storage and access methods are adopted , Efficiency will be extremely low . In turn, , A large number of large files also have corresponding problems , So in this context , Various file storage systems were born to adapt to different common , Such as ： Small files apply TFS, Large documents are applicable GridFS, also HDFS、FastDFS wait , We don't need to expand here .

Usage and frequency

The access method and frequency of saved files are usually determined by the use scenario , In practice, , We need to determine the best storage scheme according to the workload and access mode .

File access control

Whether the file can be accessed publicly ？ Can you authorize others to access ？ Whether other people in the organization have access and other full permission issues , We also need to consider more according to the actual situation , Can't ignore .

How important

In today's information society , Data and files are the most important assets , If you want to save important files , It means that in the process of storage and use, we need to do a good job in preventing loss 、 Measures to prevent damage .

The most common way to deal with loss is to leave a different place 、 Multi City 、 Multi cloud synchronous backup , Keep multiple copies .

The first half lists several key attributes , These attributes determine the selection tendency of file services , Each has its own focus , There is no detailed introduction here .

Let's take a look at how to design a file service system . After fully considering the key attributes of the document , To design a system that can be used by many different projects and teams at the same time , There are probably several options .

First option

All projects are directly connected to document services , Upload directly to the file service system from the front end of each project in one step .

This method is not intrusive to the back-end system , The front end is needed to interface with the file service . The advantage is that the file upload is done in one step , Transfer without going through the back end of the project , More efficient . The disadvantage is that the back end of the project cannot perform some customized verification and control at the first time .

Suitable for directly facing end customers , Or the use scenario of small files . for example ： Users upload personal avatars on the website , Pictures of tourism are embedded in the article .

Second option

The current end does not care about the existence of file services , The upload and use of files are directly connected with their respective backend . After uploading the file to the server of each system , Then upload to the file system through the server of each project . The advantages and disadvantages of this method and the first method are put forward , I don't remember much when uploading small files , But when uploading large files or the network is blocked , The experience will be very bad .

This scheme is more suitable for large files , Strictly controlled usage scenarios , Or the front-end system that does not need immediate feedback .

A case ： Account opening docking of wechat payment system , The merchant now sends the document directly to the service provider system , When submitting for review, the service provider system uploads it to the wechat payment system .

Key attributes and 2 There are different ways to upload , That's all for analysis .

We'll continue in the next one ！