Image processing solution veimagex technology evolution Road

​ In recent days, , The 5th deep learning image compression challenge （ Hereinafter referred to as “ CLIC  competition ”） The result of the competition was announced , The volcanic engine video cloud multimedia laboratory, which participated for the first time, won the first place in the video compression track . Compression technology for images 、 Video applications are very important . Under the premise of ensuring the same quality , How to compress the image to a smaller size for Internet information transmission , The volcano engine video cloud team continues to break through compression technology “ The ceiling ”.

At the current peak of ByteDance, it needs to process nearly one million pictures per second , Based on today's headlines 、 Tiktok and so on  DAU  Practice polishing , With internationally leading compression technology , Volcano engine video cloud creates a one-stop solution for images  veImageX , Overlay upload 、 Storage 、 Handle 、 distribution 、 Exhibition 、 Quality monitoring full link , Covering image production 、 Image consumption 、 Three application scenarios of cloud platform .

Image production

The image production scenario mainly writes the images generated by the business into the image storage , The source includes the image upload of the client 、 Storage in the mirror station or third-party cloud （ Pull on demand or full migration ）、 In the independent storage barrel of the volcanic engine 、 Pictures synthesized by the business itself .

​ Image consumption

Image consumption provides pictures  URL  pack 、 Picture real-time processing link 、 End data reporting and other capabilities .

​

Business  URL  Issue ：

During the signing process  veImageX  Issuing service , Then send pictures  URL  To the business client . Signed pictures  URL , With validity and signature information , Can effectively prevent  URL  Stealing chain 、 URL  Tampering 、 Domain name theft .

Client consumption

： Business image loading  SDK , Complete the picture download 、 decode 、 Exhibition 、 Access data and report a series of operations . Data reported by the client , After cleaning by the data reporting center , Will be stored in the data warehouse , For inquiry 、 Monitor image access performance 、 Error rate and other indicators are used .

Network distribution

： In the process of downloading pictures , Generally speaking, I will visit first  CDN . if  CDN  Not cached , Will trigger the return to the source , The request is forwarded by the source access layer to the picture stilling service . This service is mainly responsible for the verification of access rights 、 flow control 、 The main process of downloading image resources and static images . Request processing , For the need to apply image algorithm 、HEIF  Still picture  FPGA  The scene of coding , The image stilling service will pass  RPC  request  Lambda  Computing platform , The corresponding capabilities have been deployed on the platform through remote callable functions . about  FPGA  Unsupported pictures （ The resolution is too large or too small ）, It will be sent to  CPU  Platform  HEIF  Static image coding service processing . Request for dynamic image transcoding processing , The static image service will send it to the dynamic image service for processing . Due to the number of frames of the motion picture , For more frames 、 High resolution big motion pictures , When the request is processed for more than a certain time , The motion graph service will convert synchronous processing to asynchronous operation , And temporarily return the original image as the degradation result , The result of degradation is  CDN  After the cache on expires , The corresponding request will return to the source again , At this point, you can get the image after asynchronous processing . When the image still service completes the processing of the image request , Requesting the corresponding data will be stored in the data warehouse , For data analysis 、 Billing push usage .

Cloud platform

veImageX  The console acts as a one-stop cloud platform , Provide tenant management for users 、 Configure distribution 、 Usage query 、 Quality monitoring 、 Algorithm and operator management and other functions .

The request of the console is sent to  veImageX  The back-end service .

Platform management

, dependent  OpenAPI  Support platform side configuration management 、 Usage query 、 Quality monitoring and other capabilities . Scenarios involving the latter two , You need to query the data warehouse to get the corresponding data . Scenarios involving domain names , Need to rely on  veImageX  Domain name management service , Interact with various components of access and distribution .

Operator warehouse

, Image algorithm developers can easily manage 、 Release 、 Operation and maintenance picture algorithm model . Cloud data migration service is responsible for the management of data migration tasks in the picture production scene . Creative cube service realizes the ability of multi graph synthesis , Businesses can create styles through the creative cube add-on , Then dynamically replace the text 、 picture , Mass production pictures , Suitable for poster making 、 Product image synthesis and other scenes .

Offline call

, As a supplement to the real-time image distribution link , It also provides offline call of image algorithm , Contains all image algorithms  OpenAPI . In addition, the image offline transcoding analysis platform can deal with the offline transcoding of batch images 、 Image quality analysis and other scenes .

veImageX  The evolution of Technology

veImageX  The formation of these abilities is not achieved overnight , But with the business development of ByteDance, it grows all the way .

Basic general picture distribution ability

In the Internet era, business cannot be carried out without pictures , In the early stage of business , Picture distribution is under consideration . For byte skipping , The development of image distribution related systems has begun before the company was officially established .

Mainstream image transmission schemes are based on  HTTP(S)  agreement . Because the bit rate of pictures is much smaller than that of videos , For most scenes , It can be in a  HTTP(S)  Complete real-time processing in the request . Then access on the user access side  CDN , Take your own image processing service as the source station , In this way, you can cache the response of most image requests . such , Even if a basic picture distribution link is set up .

The early image system mainly solves the most basic image distribution demands of the business , And made a certain degree of abstraction on this , adopt  HTTP  The origin distinguishes different distribution scenarios . For example, use  

/obj/<store_uri>&nbsp;

It means to distribute the original drawing , Use  

/large/<store_uri>&nbsp;

It means distributing pictures of large resolution gears .

When there are more business scenarios , Personalized image display demands also follow , With  HTTP  The partial customization of route differentiation scenarios is becoming increasingly bloated . here , The picture system utilizes common practices , Classify and abstract image processing parameters , For example, cutting 、 The zoom 、 Watermark, etc . Give the choice of picture parameters to the business , Let the business send pictures  URL  Specify processing parameters in , Achieve the purpose of personalized distribution .

stay  URL  Direct mosaic image processing parameters , It can provide a great degree of freedom . But the business uses pictures more based on scenarios , Instead of parameters . Different picture distribution scenarios , Probably all use the same image processing parameters , But the picture  URL  They all look the same . At this time , Business not only needs to understand the meaning behind parameters , I can't get through  URL  To distinguish usage scenarios . in addition , With the increase of image processing ability 、 Getting complicated , The design of image processing parameters will also become a problem .

To solve this kind of problem , The technical team first defines  filter , One  filter  It can be abstracted as an image processing module , Input is  RGBA  Pixel matrix and some custom parameters , Output is  RGBA  Pixel matrix . Single  filter  It can be simple image scaling , It can also be a complex image algorithm （ Such as super score ）. Then the image processing process is abstracted into a  pipeline , Run the download in sequence -> decode -> application  filter A-> application  filter B-> code , And return the final result . With these concepts , We can define templates ： The template is a configuration file , Stored in the image system server , The image is encapsulated inside the template  filter（ Which one?  filter  What parameters ） And coding parameters . such , Businesses don't need to pay attention to internal details when using , Just click on the interactive interface , You can create your own template . In order to improve the scalability of the template , Templates also support  filter  Dynamic changes of some parameters in , Issued in  URL  In the middle of . For example, support in  URL  Dynamically specify the zoom width and height 、 Support incoming users  ID  As watermark text .

​ With the introduction of templates and more business lines , A grim fact appeared in front of us ： Where to create a management template when business access , Where to manage image resources and image domain names , How to get through the client and server ？ At this time ,veImageX  As a complete product form came into being .veImageX  Manage business resources with service as the minimum granularity （ domain name 、 Storage 、 Templates, etc ）, Many image algorithms are provided in the form of add-on components , With the usage query 、 Quality monitoring 、 Configure the linkage ability of the sending end .

HEIF  Promotion of efficient compression format

After having basic cloud image ability , As the volume of business grows ,veImageX  And business gradually pay attention to how to reduce the picture cost of business . In the picture related costs of the business ,CDN  Bandwidth costs generally occupy  80%  above . Therefore, without affecting the quality of the pictures distributed , Reducing the image bit rate is an effective way to reduce CDN  Means of bandwidth cost .

Common picture formats are  JPEG、GIF、WebP、HEIF.JPEG  and  GIF  From its launch to now, it has gone through  30  year ,JPEG  Main static diagram ,GIF  Main action diagram . Rising star  WebP  On  2010  Year by year  Google  Introduction , be based on  VP8（ And  H.264  Contemporary technology ）, At the same time, it supports dynamic and static diagrams , Same picture quality , comparison  JPEG  and  GIF  You can save  70%  Bit rate of .HEIF  The format is based on  H265 , from  MPEG  On  2015  Launched in 2013 .

Volcano engine video cloud team based on  H265  technology , With self-developed  BVC1  Codec is the core , Launched compatible  HEIF  Format efficient picture codec . at present  HEIF  The image format has covered more than  50%  Business scenario of , Use  HEIF  The compressed image bit rate can reach the same image quality  WebP  Of  55%-70%.

HEIF  Format is a double-edged sword , Compared with other formats , While improving the compression ratio , It also needs to consume more  CPU  Computing resources . In order to reduce  HEIF  Format coding calculation cost ,veImageX  Adopted  FPGA  Heterogeneous architecture . After the application , The overall cost is about  CPU  Of the plan  10%. But one of the difficulties of isomerism is ：FPGA  How to deploy the computing cluster .

There are three options .

programme  A：

The overall image processing service is migrated to  FPGA  colony ;

programme  B：

take  HEIF  Coding is independent of a service , And deployed to  FPGA  colony ;

programme  C：

The whole service division  CPU  colony + FPGA  colony , You will only need to use  FPGA HEIF  The encoded request is sent to  FPGA  colony .

​ Compared with the three schemes , programme A The most simple , But the least economical , because  FPGA  Only for  HEIF  code , Not available for other calculations . For those not involved  HEIF  Encoded request , Consume only  CPU  resources . Because on the machine  FPGA  and  CPU  The ratio is fixed , and  FPGA  Throughput compared  CPU  Much higher , This will cause  CPU  Relative shortage of resources . In order to ensure  CPU  The load is within the safe water level , here  FPGA  The load must be too low , It leads to a waste of resources . in addition ,FPGA  Resources have a lead time , and  CPU  Resources are relatively easier to obtain , programme  A  It also increases the overall operation and maintenance cost .

programme  B  The advantage of is enough flexibility , The image processing service passed  RPC  And  HEIF  Code service communication , Instead of focusing on  HEIF  Coded computing architecture . about  HEIF  Coding Services , You can go through it first  CPU  Computing architecture implementation . stay  FPGA  After the scheme is complete , As long as change  RPC  Purpose service , Seamless migration . Another independent  HEIF  Coding services use  CPU  Compared with the whole image processing service, there will be much less computing scenarios , such  FPGA  The throughput of will not be limited by  CPU  Allocation of resources , Can play to the maximum . Of course , The disadvantage is ,RPC  Call introduces additional encoding and decoding process , It is used to send the image to be encoded after processing , And a small amount of compression noise will be introduced . in addition ,RPC  With network communication overhead , The link delay becomes longer , This will offset  FPGA  Coding performance advantages of .

programme  C  There are certain requirements for image processing services and their upstream , They need to be able to control the flow , Can be needed  FPGA HEIF  Encoded requests are sent on demand to  FPGA  colony . In addition, the scheme is right  FPGA  Clustered  CPU  The ratio also has certain requirements , At this time, the computing bottleneck may be stuck  CPU  Resources , Lead to  FPGA  Throughput of is limited . The advantage of the scheme is that it can play  FPGA  The computing performance of , Reduce the source image processing delay . Of course , On the other hand , because  CDN  The existence of , When the requested quantity reaches a certain volume , The processing delay of the origin site will only affect the first brush experience of the image , It has little impact on the overall picture loading time on the end .

At present  veImageX  Yu scheme  B  And solution  C, According to different scenes , They all use ： For static scene , Use the scheme  B , because  FPGA  There is a limited range for the resolution of the picture ,CPU  and  FPGA  Two sets of computing architectures need to coexist , And the static graph request is large , Independent service maintenance enables  CPU  Not to become a bottleneck , Can dig  FPGA  The greatest potential of ; For dynamic scene , Because there are many frames in the motion picture , Code to the intermediate format and then pass  RPC  Sending costs more , programme  C  More appropriate .

Of course , The development of image format is very fast , And  HEIF  Compression performance is comparable  AVIF  Format  veImageX  Has also supported . Based on the new generation  H.266  Of  BVC2  Format is also on the way

Application of intelligent image algorithm

The algorithm used in the picture distribution scene , According to the purpose, it generally includes image quality improvement 、 Bit rate savings 、 Directional optimization ; Directional optimization mainly refers to blind watermarking 、 Cutout 、 The background is automatically erased 、 Intelligent cutting, etc .

The algorithms of image quality improvement mainly focus on super division and enhancement . Super points can provide  2-8  Times higher resolution , So as to improve the display effect of the picture , Of course, the bit rate of the picture will increase accordingly . The enhancement algorithm aims to optimize the image content , By scene, it can be subdivided into low-quality enhancement 、 Portrait enhancement, etc .

• The low-quality enhancement algorithm can deal with a variety of fuzziness 、 noise 、 Low quality image adaptive processing such as compression damage . After algorithm processing , The picture is in sharpness 、 Contrast 、 Noise and artifact elimination have been greatly improved .

The picture on the left is the original , The noise in the picture is obvious , Poor picture quality ; The picture on the right shows the results of low-quality enhancement , The overall image quality is significantly improved

• Image enhancement algorithm eliminates the blur of key face areas in the image 、 Detail recovery 、 Reconstruct clear facial features , Further improve the quality of face .

The picture on the left is the original , The portrait is vague ; The picture on the right shows the enhanced portrait , The whole portrait is clearer

The representative of rate saving algorithms is intelligence gathering and slimming . The algorithm can maintain the image quality of the original image , Reduce the image bit rate . According to the online application effect of the business , The overall bit rate saving ratio can reach  10%-15% .

These intelligent algorithms generally need to rely on  GPU  Computing resources to accelerate , This introduces heterogeneous computing resources . In practice ,veImageX  rely on  Lambda  Computing platform to manage  GPU  resources . The developer encapsulates the algorithm model as  Lambda  Functions on the platform , The image processing service passed  RPC  To apply image processing algorithms . such , The decoupling between image processing service and algorithm iteration is realized . In order to improve the efficiency of algorithm publishing ,veImageX  Built operator platform . Operator platform as  veImageX  And algorithm developers , The algorithm model is automatically put on the shelves 、 Visual operation and maintenance of algorithm functions 、 Iteration management and other functions of algorithm version .

The processing time of image algorithm is generally on the order of hundreds of milliseconds , Plus  RPC  As an intermediate result of the image file encoding and decoding and network transmission , The overall processing time may be several times that of ordinary image requests . For the first request back to the source , If image algorithm is used , The delay of access will be uncontrollable . In order to improve the first brush experience of pictures ,veImageX  For the effect promotion algorithm, the degradation method is introduced . On the first visit , Directly return the original image as the degradation result , And asynchronously call the image algorithm , Then the image processed by the algorithm is stored persistently .

​ The above discussion is in the scene of image distribution , Apply image algorithm and return in real time . Another plausible idea is , After uploading the picture , Immediately trigger the image algorithm , And save the image processed by the algorithm . When it comes to distribution , Send the processed pictures directly . This practice has some disadvantages ：

First of all, the application of image algorithm on the production side is a full-scale process , All uploaded pictures need to be processed accordingly , Over time , The storage resources occupied by this part of picture data will gradually expand . Although many pictures in it may not be accessed again , But still have to pay this part of the storage expenses .

in addition , The algorithm model will have version iteration , Some problems may be fixed 、 Or have the ability to improve , At this time, the stock of processed pictures is very “ embarrassed ”： If all stocks are replaced, use the new algorithm , The magnitude is too large , Not all of them can be handled ; If you don't replace , The new algorithm takes effect slowly , The existing problem pictures may also be irreparable .

Compared with , The advantage of real-time processing on the distribution side is reflected . Handle as needed during distribution , After the algorithm model is updated , Just replace the grayscale with the new image template , Stock and new increase can be solved together .

Control of image quality

The algorithm mentioned above , Some are to improve the image quality , Some are the premise of ensuring image quality, reducing bit rate . It all revolves around a concept ： Picture quality .

veImageX  Provides  OpenAPI, You can evaluate the image quality of the input image , From clarity 、 noise 、 Comprehensively score the pictures in terms of aesthetics . But we found that users are more concerned about the quality of the image presented by the huge collection of visited images , Instead of the image quality score of a single or several pictures . Therefore, a large market is also needed to show the user access perspective , The overall picture quality of the picture . Besides , When starting a new algorithm  AB  After the experiment , The effect of the algorithm can also be verified according to the image quality index , In this way, the results of the overall experiment are more believable .

It is definitely not a good way to directly evaluate the image quality of the displayed pictures on the client , On the one hand, image quality evaluation will consume more computing resources , On the other hand , Most of the pictures accessed by the client are cached in  CDN  Upper , It's the same , Such repeated calculation is a waste of resources . Therefore, it is more reasonable for the server to evaluate the image quality , The results of image quality evaluation can be cached in  CDN.veImageX  The image quality evaluation option has been added to the image template . If the option is turned on , After the source station processes the picture request , Add a step before returning the image data —— Image quality assessment . Since image quality evaluation is not a necessary step , Evaluation timeout or failure should not affect the return of image data .

But there is no free lunch , Image quality evaluation also involves remote algorithm calls , The overall time consumption is about... Of the original image request processing  30%-50% , Therefore, the delay of requesting the first return to the source will inevitably increase . There are two solutions to this problem ：

• Try to use it in scenarios where users visit frequently . One data for reference is ：CDN  The hit rate of edge nodes is  30%  When , If image quality scoring is introduced , The average time taken by clients to request pictures has increased by about  20% ; And when  CDN  Edge hit rate to  99%  When , The overall average time-consuming rise is less than  2% .

• Add sampling options when the source station makes image quality evaluation , Default  50%, It means only to  50%  Request for image quality evaluation . Because the data reported by the client was originally sampled （ Usually at least one in a million ）, Therefore, the sampling of the source station has little impact on the final result .

​ The online image quality evaluation scheme is mainly used for the analysis of image quality indicators in experiments and the image quality scoring of user perspectives . A problem ensued ,AB  The experiment usually takes a long time , So before applying the new algorithm to the experiment , How to evaluate the impact of algorithm on image quality , And adjust the algorithm parameters accordingly ？veImageX  Solve this problem through the offline image quality evaluation platform . The specific way is , Pull the access pictures of a group of online actual users , Use the image template containing the new algorithm for processing , In the process , For each in the template  filter  Entrance 、 Export pictures for image quality analysis , Finally, all the analysis results will be summarized into a report . In this way, according to the intermediate image quality results in the report , Can clearly evaluate the overall effect of the new algorithm . Businesses can also be exchanged according to the results  filter  The order 、 Modify the algorithm parameters .

Multiplexing of distribution links

The real-time image distribution link is just a carrier , Expand slightly , It can be applied to other scenes . for example , The original file is video , If the static diagram is issued , Then do real-time frame cutting for the video file , Extract a frame from a fixed point in time of the video file , And reuse image template processing ; If the next engine diagram , Then do real-time dynamic cover interception of video files , Take a short clip from the video file , Convert to motion pictures , Then reuse image template processing . Of course , Limited by 「 real time 」 This premise , When the original file is a small video , The effect is the best .

For real-time transcoding of small videos , It can also reuse the real-time distribution link of pictures , But it needs some customization . First of all, the video processing process is very different from the picture , Independent service processing is required 、 Independent template configuration ; Secondly, the bit rate of image processing results is generally small , It can be returned in full in a single time , But the video has to consider the viewing experience , You need to be able to support streaming return . Of course , Compared with video on demand , Just win in light access 、 Simple distribution 、 Quick to use , For many video transcoding capabilities involved in video on demand 、 Player optimization 、 Medium and long-term video support is beyond .

Some summary

veImageX  The evolution of picture system runs through the development of ByteDance , Solve various problems of picture distribution in different forms in different periods , Final “ growth ” Become the stage in the picture .veImageX  Get out of business requirements , Return to business , Close contact with business , Provides a complete set of end-to-end image solutions .

welfare ：

Click on

receive veImageX Limited time special resource package

, Get the same image processing ability of ByteDance .

​