Overview of image restoration methods -- paper notes

Link to the original text

Overview of image restoration methods

Application of image restoration

1. object removal

2. Fix image

3. Picture decoration

4. Text removal

Repair method

Traditional repair methods

“ root According to different repair ideas , It can be divided into based on partial differential equations (Partial Differential Equation,PDE) Image restoration method And sample based image restoration methods .”

（1） Method based on partial differential equation

“ Using partial differential equations in mathematics or physics cheng , The pixels in the known region of the image are smoothly propagated to the missing region to Repair the broken image .”

（2） Sample based image restoration method

“ The image restoration method based on samples is through calculation and search The sample with the highest similarity between the missing area of the damaged image and the known area , and Copy and paste it into the missing area to repair the damaged image .”

Image restoration method based on deep learning

“ According to the number of generated images , Divide it into unit image repair Complex method and multivariate image restoration method .”

“ Unit image repair Compound method refers to generating a single repaired image from a single input image ”

“ multivariate Image restoration method refers to generating multiple repaired images for a single input image ”

Comparison of four unit image restoration methods

“Transformer The performance of class repair method is better than others Three types of repair methods , The reason is Transformer[9] Models can Use the self attention mechanism to obtain a larger receptive field , Realize image distance The acquisition of information leads to the cultivation of consistent meaning and Visual Rationality Complex results .”

“Encoder-Decoder class 、U-Net Classes and GAN Class repair method to repair small missing areas (10%-40%) Broken image of It is better to , although Transformer Class repair method evaluation data set Less , But it is in the large missing area of some data sets (30%-50%) repair It still shows better repair effect than other repair methods .”

“ At present, image restoration still focuses on the study of face and scene Repair of image , While ignoring the repair of other image data sets , for example Streetscape 、 texture 、 Image data sets such as buildings .”

Research prospects

“ Next, we will focus on how to reduce the computational cost Realize the restoration of high fidelity image under the condition of 、 Of high resolution images Repair and repair of large missing areas .”

“ scene 、 The restoration of street view image is still There is a lot of room for development .”

“ There is still a large research space for damaged image restoration of large missing areas .”

Data sets

Some public image datasets and mask datasets used at present are given in this paper , The image data set contains buildings 、 texture 、 Streetscape 、 Scenes and faces .

Indicators of image evaluation

Full reference refers to selecting the original image as the reference image , Compare generated images Difference from the original image ; Semi reference refers to selecting part of the original Image as a reference , Compare and analyze the generated images ; No reference It means that the original image is not needed , Directly compare and analyze the generated images .

Future research direction

1. “ How to complete the two parts of image texture and structure at the same time ”
   Existing repairs Methods mainly include repairing only textures ( Such as MRF-Net[49])、 Just fix structure ( Such as SI[34])、 Repair the structure first and then the texture ( Such as EC[90]、 PRVS[67]) Three repair ideas . But there are certain limitations .

2. “ Research on the improvement of the performance of multiple image restoration methods and their evaluation indicators ”

3. “ Study the high-resolution image restoration model with low computational cost Is one of the most urgent tasks at present ”
   although Transformer Class repair method can realize high-resolution map Image repair and achieved high-quality repair results , But they need a lot of computing costs and expensive experimental equipment , Not suitable for business .Encoder-Decoder class 、U-Net class 、GAN Although the class repair method can also be obtained by stacking convolution Receptive field to achieve high-resolution image restoration , But stacking volumes Accumulation will also bring about an increase in computing costs 、 Fix the model Stability and so on .

4. “ How to create a dataset based on Asian face images It is the key direction of future research .”
   The current research is mainly based on the image of foreigners' faces , These image datasets train modules Type and repair Asian faces , There will be inaccurate or even wrong repairs Complex results

5. “ How to realize face restoration in different tasks and scenes , It is a difficult problem that needs to be solved urgently .”
   For example, wear a mask 、 Hair blocking 、 Face overlap and so on , These problems will improve the repair of face images difficulty .

6. “ It is a difficult problem to design an evaluation index that has no reference and can accurately reflect the image quality .”
   Currently widely used figure Like repair evaluation index MAE[131]、PSNR[134]、SSIM[135] etc. All are full reference indexes , This kind of evaluation index needs to use the original image As a reference , Calculating the pixel similarity of the whole image at the same time requires lot of time . Using the evaluation index without reference to evaluate the repaired image can save costs , Of great significance .