Image fusion GANMcC study notes

GANMcC: A Generative Adversarial Network with Multi-classification Constraints for Infrared and Visible Image FusionArticle study notes

模型结构

生成器的结构分为gradient path与Contrast path

梯度信息表示的是纹理信息
for gradient paths,连接two visible lights+an infrared image作为输入

Contrast information用亮度表示
For contrast paths,连接a visible light+Two infrared images作为输入

Indicates the probability that the input image is a visible light imagePvisand the probability that the input is an infrared imagePir.When the discriminator determines to fuse the images,The generator expects both probabilities to be large,也就是说,Let the discriminator consider that the fused image is both visible and infrared.相反,判决器Dedicated to accurately determine fused images as fake data,即Make both probabilities smaller at the same time.这样,An adversarial game is established between the generator and the discriminator.当判决器Calculate the fused imagePvis和Pir都很大时,This time is considered to be an information-balanced fused image.

生成器损失函数

生成器损失函数L-G.

其中rBalancing coefficient for balancing adversarial loss and content loss

对抗损失

对抗损失L-Gadv定义如下

其中dis the discriminator that determines the fused imageprobability label.在我们的工作中,The discriminator is an output1 × 2A multiclassifier of probability vectors.

因此,D( )[1]表示向量的第一项,即The probability that the fused image is a visible imagePvis.
同样,D( )[2]表示向量的第二项,即The probability that the fused image is an infrared imagePir.

值得注意的是,We use the same token for both probabilitiesd,So the discriminator has the same probability to determine whether the fused image is an infrared image or a visible light image.这里,Because the generator expects that the discriminator cannot distinguish between fused images and real data,所以d被设置为1.

内容损失

内容损失 由四部分组成,That is the main strength loss、main gradient loss、Auxiliary Gradient Loss and Auxiliary Strength Loss.L_con定义如下

其中β()为常数,需要进行调整,In order to realize the primary and secondary relationship between these items.The gradient loss term is usually smaller than the strength loss term,因此需要调整β()so that they are equally important in the optimization process.因此,β()The setting rules can be summarized as :

其中Major strength lossL_int-main定义如下

其中Ifusedis the fused image,可以形式化为G(Ivis,Iir),Iiris the infrared source image.

Main gradient lossL_grad-main定义如下:

Because the infrared image also has some texture details,Visible light images also contain contrast information.

因此,We propose the concept of auxiliary loss.也就是说,We construct the fusion between the image and the infrared imageAuxiliary gradient loss * Lgradaux*,and between the fusion image and the visible light imageAuxiliary strength lossLintaux,如下:

判别器损失函数

The discriminator is a multi-classifier,Its loss function must continuously improve its discriminative ability,And can effectively identify what is an infrared image or a visible light image.

鉴别器的损失函数由三部分组成,即可见光图像、红外图像和融合图像judgment loss.We denote these three losses as LDvis、LDir和LDfused.那就是:

Take into account the discriminator output1 × 2矢量,我们得到Pvis = D(x)[1]和Pir = D(x)[2].当输入是可见光图像时,预期pvi应该接近1,Pir应该接近0.The corresponding loss is defined as :

其中a1和a2为概率标签,a1设为1,a2设为0.也就是说,When inputting visible light images,There is a high probability that the discriminator wants to judge that it is a visible light image,The probability of infrared images is small.

类似地,Infrared loss term定义为:

其中b1设置为0,b2设置为1.

最后,when the input image is融合图像时,The loss function is formulated as :

其中c是The discriminator determines the probabilistic labels of the fused images,应设置为0.

同样,We also use the same label for both probabilitiesc来达到平衡.

也就是说,in the view of the discriminator,The fused image is a pseudo-visible image and a pseudo-infrared image to the same extent.

改进的点

It is heavily affected by shadows in some scenes,This results in unnatural shadow transitions in the fusion result.