Recommended papers on remote sensing image super-resolution

Supervised deep learning remote sensing methods

be based on CNN Remote sensing SR Method

2015 year ,Dong For the first time, convolutional neural network is used for super-resolution reconstruction . Because the deep convolution neural network is superior in realizing the end-to-end global optimal solution and performance .

DONG C,LOY C C,HE K,et al.Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,
38（2）：295-307.

Liebe People will SRCNN(Super Resolution Convlution Neural Network) Introduced into the super-resolution of remote sensing images . Because the ground sampling interval is large , Remote sensing image SR The mapping relationship between high and low resolution of natural images cannot be directly used , Therefore, the author uses SENTINEL2 Images （ contain 13 Frequency band , Can be up to 10 m） A remote sensing data set has been produced , Use this dataset pair SRCNN Re trained , So that it can learn the relationship between remote sensing images and high-resolution images , A network model with the ability to process multi spectral satellite images with high radiation resolution is obtained msiSRCNN（multispectral satellite images SRCNN）.

LIEBE L,KORNER M.Single-image super resolution for multispectral remote sensing data using convolutional neural networks[J]. ISPRS-International Archives of the Photogrammetry,Remote Sensing and Spatial Information Sciences,2016,41：883-890.

Due to the serious lack of local details in remote sensing images , And ordinary. CNN Super-resolution methods often use only deep features with large receptive fields SR The reconstruction , Local information is ignored ,Lei Etc. designed a “ Branch ” Structured network （Local-Global Combined Network,LGCnet）, To learn the multi-scale representation of remote sensing images , utilize CNN As the network deepens, the receptive field expands , The combination of local and global information is achieved by cascading shallow and deep feature mapping , So as to better guide remote sensing super-resolution reconstruction .

LEI S,SHI Z,ZOU Z.Super-resolution for remote sensing images via local -global combined network[J]. IEEE Geoscience and Remote Sensing Letters,2017,14（8）： 1243-1247.

LGCnet Convolution layers with different depths are cascaded , But it does not change the size of convolution kernel ,Qin In view of GoogLeNet A multiscale convolution network cascading different convolution kernels is proposed （Multi-Scale Convolutional Neural Network,MSCNN）. In the feature extraction stage, we use Convolution kernels of different sizes extract features from different angles of the image at multiple scales , After stitching the extracted features , Use deep convolution to transform this Some features are fused , Get more comprehensive depth features to improve the accuracy of the model SR effect .

QIN X,GAO X,YUE K.Remote sensing image superresolution using multi-scale convolutional neural network[C]//2018 11th UK-Europe-China Workshop on Millimeter Waves and Terahertz Technologies（UCMMT）, 2018：1-3

Although the above two methods have studied the characteristics of local information loss in remote sensing images , However, in order to ensure the fusion of feature images , The characteristic graph is filled in during convolution , So far, the feature image Contains more noise signals , It makes the model training more difficult , And it is not conducive to the final SR Image reconstruction . in addition , The above method has less network layers , While the structure of remote sensing image is complex, the amount of information is large , Therefore, the network cannot well fit the relationship between high and low resolution of remote sensing images , influence SR Accuracy of results .

Remote sensing based on residual learning SR

In order to solve the deepening of network layers , Problems leading to gradient explosion and gradient disappearance ,He etc. [33] A residual network is proposed .Pan Waiting to be received Haris Etc （Dense Back-Projection Networks,DBPN） Inspired by the , A method based on residual dense network is proposed （Residual DBPN,RDBPN）. The method in DBPN cast Dense jump connections are added to the shadow unit , Thus, global and local residuals are constructed , And through the reuse of features, it provides information for high-power amplification , Therefore, it shows better performance at high magnification .

PAN Z,MA W,GUO J,et al.Super-resolution of single remote sensing image based on residual dense backprojection networks[J]. IEEE Transactions on Geoscience and Remote Sensing,2019,57（10）：7918-7933.

Large scale super-resolution requires a lot of feature information ,pan Dense residual links are used to realize feature reuse , Dense residual links are used to realize feature reuse ,Dong Others believe that only the characteristics of the end of the network , Limit SR The effect of reconstruction , A dense sampling mechanism is designed （Dense Sampling Super Resolution, DDSR）, Transfer the characteristics of different depths of the network to the final acquisition Sample module for final reconstruction . This method is to make full use of different features , The input of channel attention mechanism is widened in the network residual block , And adopted the chain training strategy , Take the model with small scale factor as Large scale super-resolution initialization parameters to simplify training , Improve performance . This method has obvious advantages for large-scale amplification .

DONG X,SUN X,JIA X,et al.Remote sensing image super-resolution using novel dense-sampling networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021,59（2）：1618-1633

In view of the lack of remote sensing high-resolution image data , In order not to increase parameters and maintain model performance ,Haut Integrate visual attention mechanism into residual based network design （Remote Sensing Residual Channel Attention Network,RSRCAN）, This mechanism can guide the network training process towards the feature with the largest amount of information . Attention module By enhancing the high-frequency information of the image , Suppress low frequency information , So as to make the model Learn more about the mapping relationship between high-frequency components , Focus on the need for more refinement HR Details related to surface features .

HAUT J M,FERNANDEZ-BELTRAN R,PAOLETTI M E, et al. Remote sensing image super resolution using deep residual channel attention[J]. IEEE Transactions on Geoscience and Remote Sensing,2019,57（11）： 9277-9289.

Considering that attention mechanism can mine high-frequency information of remote sensing images , Zhang Et al. Proposed a high-order hybrid attention mechanism （Mixed High-
Order Attention Network,MHAN）. In the feature extraction stage Over nucleation is 1 The convolution of gives weight to different levels of convolution , Retain more important information . Frequency aware links are added in the feature refinement stage , Through the high-order attention module, the features of different depths are fused and refined , To generate richer higher-order features .

ZHANG D,SHAO J,LI.et al.Remote sensing image super-resolution via mixed high-order attention network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020,99：1-14

For remote sensing images, the scene is very different , And the size difference of the target is large problem ,Zhang Etc. using the method of transfer learning for different scenes HR and LR Remote sensing image modeling （Multiscale Attention Network, MASN）. adopt AID After the data set has trained the benchmark model , Yes 30 Fine tune the model based on the mapping relationship in the three scenarios , Finally get 30 A model , For remote sensing images of different scenes, it can adaptively allocate models for super-resolution reconstruction . In this method, a multi-level method is proposed to deal with the large size difference of the target Activate the feature fusion module , Convolution kernels of different sizes are used to extract different Characteristics of scale , The features of different scales are fused by using the channel attention mechanism . Although this method has achieved good results , But the number of models and parameters increases .

ZHANG S,YUAN Q,LI J,et al.Scene-adaptive remote sensing image super-resolution using a multiscale attention network[J]. IEEE Transactions on Geoscience and Remote Sensing,2020,58（7）：4764-4779

Although network deepening can extract more abstract features , But at present, most methods lack the use of shallow features . In order to make full use of different levels of features ,Li Etc. proposed the group parallel link module （Parallel-Connected Residual Channel Attention Network, PCRCAN）. Add several sub branches outside the main branch for feature extraction , In this way, the same features are extracted in many aspects , Improve the efficiency of information utilization .

LI Y H,IWAMOTO Y,LIN L F,et al.Parallel-connected residual channel attention network for remote sensing image super-resolution[C]//Proceedings of Asian Conference on Cpmputer Vision,2021：18-30

Although the attention mechanism in the above method can strengthen the network's learning of important features , However, there is a lack of judgment on different spatial areas with the same characteristics Sexual learning ability .Lei Etc. proposed the use of Iception Modules extract different scale features , Combine channel attention and spatial attention mechanism , Learn to distinguish important features of the network （Inception Residual Attention Network,IRAN）, Then, we allocate attention to different areas of each feature map . This method can carry out discriminative learning of remote sensing features comprehensively , But it increases the complexity of the model .

LEI P,LIU C. Inception residual attention network for remote sensing image super-resolution[J]. International Journal of Remote Sensing,2020,41（24）：9565-9587.

SZEGEDY C,LIU W,JIA Y,et al.Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2015：1-9. Put forward Iception modular .

The above methods based on deep learning are all reconstructed in the spatial domain , And committed to learning LR And HR The relationship between the corresponding pixels in the image , Few studies have applied the frequency domain method to deep learning . Regarding this ,Ma Wavelet transform is introduced into remote sensing super-resolution based on depth learning , A residual recursive network combined with wavelet transform is proposed （Wavelet Transform Combined with Recursive Res-net,WTCRR）. The model is deleted DRRN（Deep Recursive Residual Net-work） Online BN Layers get . This method uses wavelet transform to decompose the image in frequency domain , The high-frequency component obtained by wavelet decomposition and the original image are used as the input of the network SR mapping . according to WTCRR Jump links in , Every module in the network can use the original features , More comprehensive depth features can be obtained at the end of the network to carry out the final inverse wavelet transform SR The reconstruction

MA W,PAN Z,GUO J,et al.Achieving super-resolution remote sensing images via the wavelet transform combined with the recursive res-net[J]. IEEE Transactions on Geoscience and Remote Sensing,2019,57（6）：3512-3527

Although the super-resolution method based on residual learning can effectively improve SR Accuracy of results , But the network structure is more complex , Longer training time , High dependence on data , It is difficult to achieve good results in the absence of data .

Supervised remote sensing based on generating countermeasure network SR Method

SRGAN It is designed for super-resolution GAN（Generative Adversarial Network）,SRGAN Being able to take advantage of perceived loss will SR The result is pushed to the manifold of natural image , To get an image that is more in line with human visual perception , With the help of GAN Our training strategy can make SR Generator generation Into an image more in line with human perception . Whereas SRGAN Excellent results achieved , The following methods are used GAN Thought , Make the super-resolution effect of remote sensing image more consistent with the observation effect of human eyes .

The degradation process of remote sensing image is affected by many factors , It often contains more noise , And based on GAN The original method of is sensitive to noise , It will produce high-frequency noise independent of the input image . In land cover classification , In advanced computer vision tasks such as ground target recognition , This problem will reduce accuracy . Regarding this ,Jiang From the perspective of edge enhancement , Put forward EEGAN（Edge-Enhanced GAN）. First, the improved dense residual blocks are used to form the generation network ; Generate SR After the reference image ; adopt Laplacian Edge enhancement network constructed by operator , extract SR Reference image And enhance it ; After getting the enhanced edge SR Reference image Fusion produces a clear edge HR Remote sensing image , So as to alleviate the problem of blurring the edge of the target ground object in the remote sensing image .

JIANG K,WANG Z,Yi P,et al.Edge-enhanced GAN for remote sensing image super-resolution[J].IEEE Transactions on Geoscience and Remote Sensing,2019,57（8）： 5799-5812.

Compared to natural images , Remote sensing images have more flat areas and more low-frequency image components , Use GAN The remote sensing image is SR when , It is difficult for the discriminator to judge whether these image regions are real HR Remote sensing image Generated or sampled （ Distinguish fuzzy problems ）, Cause to generate HR remote Image quality is affected . Regarding this Lei And so on GAN The Internet ： Coupling identification GAN（Coupled-Discriminate GAN, CDGAN） The Internet ; By building a dual channel network, it will be true HR Images and SR The image is input to the discriminator at the same time , The features extracted from the dual channel network are spliced and input into the subsequent layer , A special coupling loss function is constructed to update the network parameters , The model enhances the discrimination of low-frequency regions of remote sensing images , Improved based on GAN Image SR Methods when dealing with low-frequency image areas, the phenomenon of blur is resolved .

LEI S,SHI Z W,ZOU Z X.Coupled adversarial training for remote sensing image super-resolution[J]. IEEE Transactions on Geoscience and Remote Sensing,2019, 58（5）：3633-3643.

Yu Such as to DBPN Build a generator network based on E-DBPN（Enhanced-DBPN）, And an improved residual channel attention mechanism is added , In order to promote the network to have the discriminative learning ability of features , So as to keep right SR More contributing features . meanwhile , A sequential feature fusion module is designed , Deal with the upper projection unit in the form of progressive fusion
Feature mapping of . This method takes advantage of DBPN The error feedback mechanism of LR and HR Explore the deep relationship between , Strengthen with confrontation generation strategy Of the model SR performance .

YU Y,LI X Z,LIU F X,E-DBPN：Enhanced deep back-projection networks for remote sensing scene image superresolution[J].EEE Transactions on Geoscience and Remote Sensing,2020,58（8）：5503-5515.

HARIS M,SHAKHNAROVICH G,UKITA N.Deep backprojection networks for super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2018：1664-1673. Put forward DBPN

be based on GAN Supervised learning methods , It can effectively improve the generation Of HR Perceived quality of image , But with the help of discriminator network , Increased the difficulty of training . In the above method EEGAN The edge of ground objects in remote sensing images is strengthened , Information about flat areas is ignored ,CDGAN[48] Enhance the relatively flat area in the remote sensing image , Ignoring edge information , and E-DBPN Lack of discriminant processing of spatial information of remote sensing images , The ability to restore tiny details in remote sensing images is insufficient . Above 3 Kind of Methods only enhance one feature of remote sensing image , In practice , There are certain limitations .

Remote sensing based on deep learning SR Unsupervised methods

The network structure of supervised learning realizes from LR Image to HR The mapping of images , However, high-resolution remote sensing images are usually difficult to obtain , And the degraded image is still different from the actual low resolution remote sensing image , The unsupervised method can be applied to each specific... Without using any other external data LR Input image for super-resolution processing ,Haut From the perspective of unsupervised remote sensing data SR The reconstruction , To build the SR remote The generation network model of perceptual image （A New Deep Generative Network,ANDGN）. This method first expands the random noise to the target HR dimension , Supplement image information by generating network , Generated by network HR Results after down sampling and original LR An iterative loss function is constructed for remote sensing images to ensure generation HR Image and LR The image corresponds to , In subsequent iterations, the generated image will be used as network input , Through repeated iterations until the final required HR Remote sensing image .

HAUT J M,FERNANDEZ-BELTRAN R,PAOLETTI M E,et al.A new deep generative network for unsupervised remote sensing single-image super-resolution[J]. IEEE Transactions on Geoscience and Remote Sensing,2018,56（11）：6792-6810.

Wang Based on CycleGAN This paper presents a method for remote sensing image SR Unsupervised learning network CycleCNN, The network includes image degradation network and image super-resolution network . In degenerate Networks GaoFen-2 in GSD by 1 m/pixel Panchromatic image as HR Degradation ; In super-resolution Networks , take GSD by 4 m/pixel Multispectral （MS） In front of 3 Bands are converted to YCbCr Color space , Used as a LR Perform super-resolution reconstruction . The images obtained from the degenerate and super-resolution networks are input into the super-resolution and degenerate networks respectively , Build a cycle loss function , Thus, the degenerate network can degenerate a more realistic low resolution image , Improve the performance of super-resolution network .

WANG P,ZHANG H,ZHOU F,et al. Unsupervised remote sensing image super-resolution using cycle CNN[C]//IEEE International Geoscience and Remote Sensing Symposium,2019：3117-3120.

stay Haut Inspired by ,Zhang And so on proposed a kind of based on GAN Super-resolution network model of unsupervised remote sensing image based on （Unsupervised GAN,UGAN）. The network directly uses LR The remote sensing image is used as the input of the generation network , The convolution layer with gradually decreasing kernel size is used to extract features of different scales , For unsupervised SR Retain more information , And by calculating each image and its high-level features L1 Loss and SSIM Loss , Improved loss function .

ZHANG N,WANG Y,ZHANG X,et al. An unsupervised remote sensing single -image super-resolution method based on generative adversarial network[J]. IEEE Access,2020,8：29027-29039.

In the above method ,Wang Data with different sampling intervals Sets are used as inputs to generators and degenerate models to simulate HR and LR Images , There is still a certain gap with the real data .Zhang Using average pooling , Although it can improve the generalization ability of the degradation model , But the impact is far away There are many factors of image degradation , Pooling cannot completely represent the degradation model . For the complex degradation of remote sensing images ,Zhang Et al. Proposed to train a degradation model with a large number of synthetic degradation data sets , Then, a generator model is designed to make the original remote sensing image first SR Then degenerate , With degraded results and original LR The loss function optimization network is formed by the difference of remote sensing image . The network （Multi-Degradation GAN,MDGAN） It can truly simulate the degradation process , However, there are still great differences between remote sensing image degradation and natural image degradation , Simulation of remote sensing image degradation requires consideration of imaging 、 Atmosphere and other factors , Modeling is more difficult than natural images .

ZHANG N,WANG Y,ZHANG X,et al.A multi-degradation aided method for unsupervised remote sensing image super resolution with convolution neural networks[J]. IEEE Transactions on Geoscience and Remote Sensing,2020（1）：1-14.

Remote sensing based on deep learning SR Methods can be learned directly LR Remote sensing and HR The relationship between remote sensing images . But in supervised remote sensing SR In the method , Model training requires manual down sampling of the original image to obtain training data , The data obtained by manual down sampling SR And after The loss function is established by making a difference in the original remote sensing image . Such a model is unreasonable , What exists in reality LR Remote sensing image and HR The relationship between remote sensing images is complex , Only manual down sampling is used to simulate ,SR The effect is limited . And unsupervised remote sensing based on deep learning SR Method performance is not significant , Compared with the supervised method, there is a big difference in effect .

Direction

1. original HR The problem that remote sensing images are difficult to obtain .SR The task needs to start from LR Image reconstruction HR Images , At present, most of the methods use the HR-LR Train and test your performance . However , Due to the atmospheric disturbance in the process of remote sensing image acquisition 、 Effects of noise and motion , Actual HR Images are hard to get . therefore , The current method is still far from practical use . at present , be based on GAN The network method tries to overcome this difficulty by using generators , Some unsupervised methods try to get by learning the degradation process HR Images . Considering that transfer learning can learn a priori from other samples , And optimize in the target domain , We can learn from the methods of transfer learning and zero sample learning to solve this problem .
2. Remote sensing images are more natural , The loss of details is more serious . Usually , The actual distance represented by a single pixel of the remote sensing image exceeds 5m, Cause the loss of details in the image , Reconstruction is difficult . Regarding this , The existing methods are from local - Global joint feature extraction 、 Attention mechanism 、 Combined with wavelet transform 、 Start with strategies such as edge enhancement , Try to reconstruct richer details . You can refer to the method of small target detection , Extract and enhance the subtle target objects in the remote sensing image , To alleviate the lack of details .
3. The content of remote sensing image scene varies greatly . Remote sensing photography often involves a variety of landforms , Therefore, a variety of scene contents will be photographed , Lead to the diversity of samples . therefore , The contribution of remote sensing data sets to a single scene is also weakened . This requires the designed method to be more inclusive of samples , Learning ability is stronger , That is, a model is used to describe the target in different scenarios HR And LR The relationship between . In response to this problem , The literature [39] Yes 30 Three different remote sensing scenes are modeled by super-resolution , But it takes a lot of planning Calculate and store costs . remote sensing SR Model lightweight is of great significance for multi scene modeling .ZHANG S,YUAN Q,LI J,et al.Scene-adaptive remote sensing image super-resolution using a multiscale attention network[J]. IEEE Transactions on Geoscience and Remote Sensing,2020,58（7）：4764-4779.
4. Remote sensing image under the same scene , The space size of the target objects varies greatly . A remote sensing image often contains multiple targets , And the sizes of these objects are different , For example, in the street scene, the vehicle may only occupy a few pixels , The house occupies hundreds of pixels , In the process of convolution , The characteristics of small targets may be lost , Thereby affecting SR Accuracy of results . because and , Multilevel feature extraction and feature fusion in remote sensing SR The field still has important research significance .