Image fusion DDcGAN study notes

DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion Article Study Notes

Feature: Dual Discriminator

Model Structure

The whole process of DDcGAN is shown in the figure.Given a visible light image v and an infrared image I, our ultimate goal is to learn a generator G conditioned on them, and the generated images G(v, I) are encouraged to be sufficiently realistic and informative todeceive the authenticator.

Meanwhile, we utilize two opposing discriminators, Dv and Di, which each generate a scalar that estimates the probability of input from real data rather than G.

Specifically, Dv aims to discriminate generated images from visible light images, while Di is trained to discriminate between original low-resolution infrared images and downsampled generated/fused images.Average pooling is used here for downsampling, since average pooling preserves low-frequency information compared to max pooling, and thermal radiation information is mainly presented in this form.In other words, for balance between generator and discriminator, we do not feed source images v and I as additional/conditional information to Dv and di other than the input to the discriminator.That is, the input layer of each discriminator is a single-channel layer containing the sampled data, rather than a two-channel layer containing the sampled data and the corresponding source image as conditional information.Because when the conditions and samples to be discriminated are the same, the discrimination task is simplified to judge whether the input images are the same, and this is a simple enough task for the neural network.When the generator cannot fool the discriminator, the adversarial relationship cannot be established, and the generator will tend to generate randomly.

Difference between PG and two true distributions (i.e. PV and PI)through the adversarial process of generator G and two discriminators (Dv and Di)will simultaneously become smaller, where PG is the probability distribution of the generated samples and PV is the true distribution of the visible image>, PI is the true distribution of the infrared image.

Generator loss function

GeneratorLoss function

where adversarial loss is defined as

where loss of content is defined as

The discriminators in DDcGAN, namely Dv and Di,It plays the role of discriminating the source image and the generated fused image.The adversarial loss of the discriminator can compute the JS divergence between distributions to identify whether intensity or texture information is inauthentic, thereby encouraging matching the true distribution.

Discriminator loss function