Overview of super-resolution reconstruction of remote sensing images

Overview of super-resolution reconstruction technology of remote sensing images based on deep learning

The abstract part

The methods of remote sensing image super classification reconstruction based on deep learning are divided into three categories

Super fractional reconstruction of a single remote sensing image

The best is based on GAN Methods

Method ：

1. Method based on multi-scale feature extraction
2. Combined with wavelet transform
3. Hourglass like method of generating network
4. Methods of edge enhanced network
5. Cross sensor method

Hyperspectral reconstruction of multiple remote sensing images

Poor super score effect , There is registration fusion 、 Problems such as multi-source information fusion

many / Hyperspectral remote sensing image super fractional reconstruction

Poor super score effect , There is registration fusion 、 Problems such as multi-source information fusion

Future trends

1. Build a neural network structure according to the characteristics of remote sensing images
2. Unsupervised learning method for super segmentation reconstruction of remote sensing images ( idea ：GAN It can be used for unsupervised learning )
3. Hyperspectral reconstruction method of multi-source remote sensing images

introduction

Concept

Super resolution reconstruction of remote sensing image is the processing of low resolution remote sensing image with complementary information , To obtain high-resolution remote sensing images .

Methods the framework

The origin of the method of hyperspectral reconstruction of remote sensing images based on deep learning

Traditional remote sensing image super division reconstruction method

1. Frequency domain method

   It is mainly divided into spectrum anti aliasing Algorithm , Recursive least square method, etc , Its The theoretical premise based on is too idealistic , Cannot be effectively applied to Multiple occasions , So at present, this kind of method has no longer become the mainstream of research .

2. Airspace method

   Although the airspace method is flexible , But it involves intra frame motion Fuzzy 、 Many factors such as optical blur and other complex degradation models , The optimization method is complicated , The cost of calculation is high , Failed to become the mainstream method .

3. frequency domain - The method of airspace combination

   frequency domain - The method of airspace combination , Although it combines the advantages of frequency domain method and spatial domain method , But the calculation is complex , Huge amount of calculation , Nor can it become the mainstream method .

The steps of super-resolution reconstruction of remote sensing image based on deep learning

Put the low resolution and high resolution remote sensing images into the depth learning method , Learning and training , Get the relationship between low resolution and high resolution remote sensing images . Then send the low resolution remote sensing image into the trained depth model , Get the super-resolution remote sensing image after reconstruction .

Super resolution reconstruction of single remote sensing image based on deep learning

be based on CNN Super resolution of a single remote sensing image

Commonly used CNN framework

SRCNN

The first will CNN Methods applied to super-resolution problems .2016 year Liebel Wait for the first time to propose SRCNN It is used for super-resolution reconstruction of a single remote sensing image . The method The characteristics of scale diversity of remote sensing images are not considered , So the reconstruction effect The fruit is not obvious , So people based on the scale diversity characteristics of remote sensing images , Many methods based on multi-scale feature extraction are proposed .

VDSR

stay SRCNN The residual network is introduced on the basis of , Expanded SRCNN The function of . Because remote sensing images are very different from natural images , Objects and environments contained in remote sensing images are often coupled , and And the image scale span is large , But at present, most networks are Build dictionaries or prior knowledge within a single object , Ignoring environmental information . therefore ,2017 year ,Lei A new method based on VDSR Network of , It's called local - Integrate the network globally ( local-global combined networks,LGCNet).

The network is divided into three parts ：

1. Feature extraction part , utilize L Layer convolution , Get the feature representation of different layers
2. Local - Global information integration part , Feature representation of different layers ;② Local - Global information The combination part , This part is also the whole LGCNet The core of , Mainly cascade the results of different layers , It's called “ Multi fork structure ”
3. Reconstruction part , Reconstruct high-resolution remote sensing images
   

2018 year ,Xu Etc. also aims at the problem of feature extraction at multiple scales , Propose deep memory connection network ( deep memory connected network,DMCN). DMCN It is a symmetrical hourglass structure CNN, With multiple jump connections . And LGCNet comparison , Although its reconstruction effect is only slightly improved , But the memory consumption and test time are greatly reduced .

Although the above two works use multi-scale feature extraction Methods , But the network is deep and simple , Unable to flexibly adapt to multi-scale features , The network is deep and simple , Unable to adapt flexibly Multiscale features , therefore 2020 year , Wang Et al. Proposed a super-resolution adaptive multi-scale feature fusion network for remote sensing images . This method proposes an adaptive multi-scale feature extraction module block , This module combines squeeze excitation network and adaptive gating mechanism . Through this module, the extracted multi-scale information can be fused adaptively , This method is better than SRCNN、ESPCN、MSRN And other classic methods .

EDSR

It is the best performance of general image super segmentation reconstruction at present One of the ways .

Super resolution of remote sensing image combined with wavelet transform

2019 year ,Ma Et al. Proposed another method combining wavelet transform with local 、 Global residual connected CNN Super resolution network structure (WTCRR). The network has local and global residual connections , Solve the super-resolution problem by focusing on the frequency domain .

The network has three parts ：

1. decompose

   Will pass through double 3 Sub interpolated image passes 2D Discrete small The wave transformation is decomposed into 4 A low resolution image

2. forecast

   Break down the previous step 4 Image input to has recursion and Residual network CNN In the network structure , Get preliminary prediction results

3. The reconstruction

   The picture obtained from the prediction part is discretized by inverse Wavelet transform for combination , Reconstruct the final high-resolution image .

And DRNN( deep recurrent neural network) comparison , The edge of the reconstructed image is improved .

Super resolution of remote sensing image with high magnification factor

at present , Appropriately enlarge the spatial resolution of a single remote sensing image ( for example ×2,×3) The problem of has been effectively solved , But Gao Amplification factor ( for example ×4,×8) Super resolution reconstruction of remote sensing image
It's still a problem .2019 year ,×4,×8) Super resolution reconstruction of remote sensing image It's still a problem . Pan Aiming at the above problems, a fast residual dense back projection network is proposed ( fast residual dense back projection network,FRDBPN). FRDBPN Adopt up projection and down sampling unit including specific convolution and extended convolution sequence , And global and local residual connections .

be based on GAN Super resolution of a single remote sensing image

Hourglass shaped generation network of remote sensing image super-resolution

2018 year ,Haut Focus on generating models , Propose an hourglass shaped generation network architecture . The network operates in an unsupervised manner Training . The generated network is input by random noise , Through mining Sample generates low spatial resolution data , And pass it to the upper sampling structure , After repeated optimization, a high-resolution image is finally generated . Zoom in on the image 4 Time doubling effect is better .

Super resolution of remote sensing image based on edge enhancement network

Due to noise pollution , Remote sensing image restoration high-frequency edge fine Section still needs to be improved . therefore 2019 year ,Jiang Et al. Proposed an edge enhancement network based on generating confrontation network , The generation network of the network consists of two sub networks , Ultra dense subnet ( ultra dense sub network, UDSN ) and edge edge increase strong Son network ( edge enhancement subnetwork,EESN). When UDSN For heavy When creating images , The reconstruction results look sharp , The edge is polluted by noise Seriously infected , therefore , Specially designed EESN By learning noise masking Code to enhance the edge . This method is better than classical SRCNN、VDSR[10] and SRGAN The reconstruction effect of this method has been greatly improved .

Super resolution of remote sensing images that can span sensors

The data used for super-resolution reconstruction of remote sensing images is usually limited Specific location and specific sensor , Therefore, the ability to generalize the model between different positions and sensors is limited . therefore 2020 year ,Xiong After modification SRGAN Loss function and network structure , And put forward some suggestions for improvement SRGAN( improved SRGAN,ISRGAN), This method makes the model training more stable , And enhanced Generalization ability , Realize a training of remote sensing images , It can be used in any place and different sensors . It turns out that , Use ISRGAN The accuracy of land cover classification after super-resolution reconstruction Accuracy can be significantly improved , Especially on roads and buildings with high-resolution textures , Improved accuracy 15%.

Super resolution reconstruction of multiple remote sensing images based on deep learning

Compared with super-resolution reconstruction of a single remote sensing image , There are more complementary information in the super-resolution reconstruction of multiple remote sensing images , The process is also more challenging , Additional challenges such as joint registration and fusion between multiple low resolution images need to be solved .2020 year ,Deudon Wait for... To be put forward HightRes-net The Internet , For the first time, the method of deep learning is introduced to Super resolution reconstruction of multiple remote sensing images , This method learns subtasks in an end-to-end way .

Subtasks include

1. Image registration
2. Image fusion
3. On the sampling
4. Registration loss function part
   

HightRes-Net It is a deep structure , By referring to Test frame channel , Learn to jointly register any number of low resolution images , Then up sampling and super-resolution reconstruction .

What you will get in the end ShiftNet Loss feedback to HightRes-Net In the model , Get the reconstructed picture .ShiftNet damage The loss function ensures that the super-resolution reconstruction of multiple remote sensing images will not be due to insufficient registration , Subpixel offset and so on lead to blurred result output .

Many based on deep learning / Super resolution reconstruction of hyperspectral remote sensing images

The spatial division of sensory images Reconstruction method of resolution and spectral resolution . Spectral image contains number Hundreds of bands , It provides rich spectral characteristics , Each image represents the reflectivity and emissivity of the scene in different electromagnetic bands , This kind of thing Rational representation plays a crucial role in many applications , for example Remote sensing classification 、 Target detection and anomaly detection . But because of the hardware Limit , It is more difficult to collect high-quality hyperspectral images than panchromatic images , And the resolution is also low , Therefore, super-resolution reconstruction technology has also been introduced into this field . Here are some representative methods .

be based on SRCNN More / Super resolution reconstruction of hyperspectral images

2017 year ,Yuan And so on, which will be used in the single image super division reconstruction method pre trained on natural images SRCNN Used to enhance more independently / Each frequency band of hyperspectral remote sensing image , And use the idea of transfer learning for reference to transform it into hyperspectral images . in addition , In order to study low scores Discrimination and more / Spectral properties between high-resolution hyperspectral images , The authors also propose a cooperative nonnegative matrix decomposition to enhance the correlation between low resolution and high resolution spectra .

[1] He Z ,  Liu L . Hyperspectral Image Super-Resolution Inspired by Deep Laplacian Pyramid Network[J]. Remote Sensing, 2018, 10(12).

This article also considers similar Yuan The idea of et al. First trains the Laplace pyramid network with natural images, and then uses the pre trained network to many / Each band of hyperspectral remote sensing image is super-resolution The reconstruction ; Last , Using coded spectrum in low spatial resolution more / Training on hyperspectral images , To improve more / Hyperspectral resolution .

Improve the number of loss functions / Super resolution reconstruction of hyperspectral images

Most are based on CNN Image super-resolution reconstruction network , All use the default L2 Loss function , But for hyperspectral images , This loss function may cause spectral inconsistencies . many / high Hyperspectral of spectral images , The loss function should not only focus on spatial information , Also pay attention to spectral information . therefore ,2018 year , You should only pay attention to space information Rest , Also pay attention to spectral information . therefore Zheng Et al. Proposed a multi loss function network considering both spatial and spectral information , It consists of a series dense residual network ( concatenate dense residual network,CDRN) And loss network ( loss network,LN) form . CDRN It is the image reconstruction part ,LN Including spatial loss and spectral loss of pixels , It drives the learning of the whole model .

Related information

Data sets

1. UC Merced Data sets , It is used for research Study the data set of land use , Altogether 2 100 A picture , contain 21 Class scenes , Every kind 100 Zhang , The pixel size of each picture is
   256×256.
2. RSCNN7 Data sets , It includes 2 800 A remote sensing image , They are from 7 There are different scene categories , Each category contains 400 Zhang image .
3. NWPU-RESISC45 Data sets , From Northwestern Polytechnic University , Total 31500 A picture , It is divided into 45 Scene categories , Each kind has 700 Zhang , The size of each image is
   256×256.
4. Kaggle Open source datasets , Is an open source dataset , The dataset Draper The satellite image contains 1 000 Several high-resolution aerial photos taken in Southern California .
5. Sentinel-2 Data sets , It is Sentinel In the series One of the series , Data is free , The main payload is multi light Spectral imager , share 13 Band , The spectral range is 0. 4 ~ 2. 4 μm Between , It covers visible light 、 Near infrared and short wave infrared , And the data set is used as Landsat The supplement of the system in the field of earth observation , At present, there are more and more applications .

Result analysis

For single remote sensing The results of image super-resolution reconstruction method show that , Using a GAN Structure , And pre training on the general image ,PSNR Improve higher , It can achieve the best reconstruction effect at present . Yes Yu Duo / Hyperspectral remote sensing image super division reconstruction method ,PSNR promote Significantly lower , This is mainly due to more / Characteristics of hyperspectral remote sensing images Caused by particularity , This result shows that , Using both spatial and spectral information , You can get better performance .

Discuss

1. In terms of model types , be based on CNN The method is still At present, it is the main method of remote sensing image hyperspectral reconstruction based on deep learning Flow method , However, there are also a few scholars who apply other depth models to the hyperspectral reconstruction of remote sensing images .
2. From the single remote sensing image reconstruction method and its effect see , be based on GAN This method can achieve the best reconstruction effect at present , This is consistent with the results of the method on natural images .
3. For super-resolution reconstruction of multiple remote sensing images , Additional challenges such as image registration and fusion need to be considered , So rebuild The model will be relatively large , And at present, based on deep learning, multiple remote There are few super-resolution reconstruction methods for sensory images , Mainly for single remote sensing image reconstruction methods .
4. For many / Super resolution reconstruction of hyperspectral remote sensing images The situation of , The best way to improve the performance is to improve the spatial resolution and spectral resolution at the same time , Instead of just considering spectral channels independently, as in earlier methods .
5. Most of the remote sensing image super segmentation based on deep learning The resolution reconstruction network uses L1 and L2 Loss function , These two All loss functions are based on pixel differences . For many / Spectral resolution reconstruction of hyperspectral remote sensing images , It is necessary to design an appropriate spectral information loss function , Only in this way can we generate high-quality multi / Hyperspectral remote sensing images .

expectation

1. Study the neural network structure according to the characteristics of remote sensing images . Remote sensing image has There is scale diversity 、 The particularity of perspective 、 Multi direction and background complexity
   Advanced features .
2. This paper studies the super-resolution reconstruction method of remote sensing image based on unsupervised learning . Remote sensing images with different resolutions in the same scene Images are hard to get , So there is a lack of training samples , And at present, it is mainly Our research is a supervised super-resolution reconstruction method of remote sensing images , However, the model trained by the super score method with supervised learning cannot Well deal with the super sub tasks in some actual scenes , So there is no prison Du's remote sensing image super fractional reconstruction model has important research value .
3. This paper studies the super division reconstruction method of multi-source remote sensing images . Fuse the image information of different sensors in the same scene ( Including spatial information 、 Time information 、 Spectral information and location information ), At present, the base In the deep learning method of remote sensing image super division reconstruction , Most of them are reconstructed based on one kind of information , The information learned by the model Less , Limited reconstruction effect , Therefore, research on multi-source remote sensing image information The integration of , Fully tap these complementary information , To improve remote sensing Image reconstruction effect , This will also be a very important research direction in the future .

reference ：[1] Cheng Keyan , Ronglan , Jiang Senlin , Zhan Yongzhao . Overview of super-resolution reconstruction technology of remote sensing images based on deep learning [J/OL]. Journal of Zhengzhou University ( Engineering Edition ):1-9[2022-06-21].DOI:10.13705/j.issn.1671-6833.2022.05.013.