FusionGAN: A generative adversarial network for infrared and visible image fusion article study notes

Generative Adversarial Networks for Fusion of Infrared and Visible Light Images

Main contributions to the article

We propose a generative adversarial architecture and design a loss function specifically for infrared and visible image fusion.The feasibility and superiority of GANs for image fusion are discussed.To the best of our knowledge, this is the first time a genetic algorithm is employed to solve the task of image fusion.

The proposed FusionGAN is an end-to-end model in which fused images can be automatically generated from input source images without the need to manually design activity level measurements or fusion rules.

We conduct experiments on publicly available infrared and visible image fusion datasets and compare qualitatively and quantitatively with state-of-the-art methods.Compared with previous methods, the proposed FusionGAN can obtain clear infrared images with clear salient objects and rich textures.

The proposed fusion method generalizes to fusing source images of different resolutions, such as low-resolution infrared images and high-resolution visible images.It produces high-resolution resulting images that are not affected by noise caused by upsampling of infrared information.

Model Structure

First, we connect the infrared image Ir and visible light in the channel dimensionImage Iv.Then, the concatenated images are fed into the generator G, whose output is the fused image If.Due to the loss function of the generator designed in this paper, in the absence of the discriminator D, If tends to preserve the thermal radiation information of the infrared image Ir and the gradient information of the visible light image Iv.After that, we feed the fused image If and the visible light image Iv to the discriminator D, which aims to distinguish If and Iv.The proposed FusionGAN establishes an adversarial game between the generator G and the discriminator D, If will gradually include more and more detailed information in the visual image Iv.In the training phase, once the generator G generates samples (i.e., If) that the discriminator D cannot distinguish, we can get the desired fused image If.

Loss function

The loss function of the generator includes the adversarial loss L_adv between the generator and the decider and the content loss L_content between the generated image and the real image, Generator loss functionDefined as follows:

where α is balanced confrontationBalance factor for loss and content loss.L_adv and L_content are defined as follows:

Among them, I_f^n represents the nth fused image in the same batch of N, and D(∙) represents the output of the decider.

where H and W represent the height of the input image, respectivelyand width, ‖∙‖_F represents the matrix Frobenius norm, ∇ represents the gradient operator.The first item of L_content aims to preserve the thermal radiation information of the infrared image I_r in the fusion image I_f, and the second item of L_content aims to preserve the gradients contained in the visible image I_vinformation, ε is a positive parameter that controls the trade-off between the two terms.

Decision:

where a andb represents the label of the fused image I_f and visible light image I_v, respectively, and D(I_v ) and D(I_f ) represent the classification results of the visible light and fused image, respectively.
The discriminator is designed to discriminate between fused and visible images based on features extracted from visible images.We use a least squares loss function, which follows minimizing the Pearson χ2 divergence.This makes the training process more stable and the discriminator's loss function converges faster.

Experimental results in the text

From left to right: fusion results of infrared image, visible light image, classical method, and fusion result of FusionGAN.
The third image is the fusion result obtained by using the classic method.Obviously, this traditional method can only preserve more texture details in the source image, while the high contrast between the target and the background in the infrared image cannot be preserved in the fusion image.In fact, the key information in the infrared image (i.e. the thermal radiation distribution) is completely lost in the fused image.
The far right image is the fusion result of FusionGAN.In contrast, our results preserve the thermal radiation distribution in the infrared image, so the target can be easily detected.At the same time, background details (i.e. trees, roads, and aquatic plants) in visible light images are also well preserved.
FusionGAN results can keep both the thermal radiation distribution in the infrared image and the apparent texture in the visible image.

Summary of other bloggersExcellent image fusion algorithms in the past five years