[paper reading] unpaired image to image translation using cycle consistent advantageous networks

[ Address of thesis ][ Code ][ICCV 17]

Abstract

Image to image translation is a kind of visual and graphic problems , The goal is to use training of a set of aligned image pairs to learn the mapping between input images and output images . However , For many tasks , Paired training data is not available . We propose a way , Learn to remove images from the source domain without pairing instances X Translate to the target domain Y. Our goal is to learn a mapping $G：X->Y$, send $G(X)$ Image distribution and use of antagonistic loss $Y$ The distribution is indistinguishable . Because this mapping is highly under constrained , Let's map it to a reverse $F:Y->X$ Combine , And introduce a cyclic consistency loss to promote $F(G(X))X$( vice versa ). Qualitative results are presented on several tasks that do not have paired training data , Including the transfer of collection style 、 Object conversion 、 Shift of seasons 、 Photo enhancement and so on . The quantitative comparison with several previous methods shows the superiority of our method .

Method

This article is famous CycleGAN, The core idea of the method is as follows ：

It consists of two generators $(G,F)$ And two discriminators $(D_X,D_Y)$ constitute . For the input source domain image $X$, Send it to the first generator $G$, Then you can get a false target domain image $G(X)$. Judging device $D_Y$ Need to be able to distinguish the actual target domain image $Y$ And false target domain images $G(X)$, So that the generated $G(X)$ The style features included are more ; At the same time , Swap the target domain with the source domain , Then the target and image $Y$ Sending a generator $F$ after , You can get a fake source domain image $G(Y)$. Judging device $D_X$ You need to be able to distinguish the actual source domain image $X$ And fake source domain images $G(Y)$, So that the generated $G(Y)$ The style features contained are more realistic .

The advantage of this is , Because the task of image style conversion in this paper is " Unsupervised ", No matching " From the - Target domain " The image is right , It is equivalent to only being able to constrain whether the generated image meets the new style , There is no way to constrain whether the generated image is consistent in content . And with cycle After the form , After a picture goes in , First, it becomes $G(X)$, And then it becomes $F(G(X))$, By restraint $X$ Should be the same $F(G(X))$ As similar as possible , So as to ensure that the network still needs to maintain details as much as possible while learning how to change styles , To achieve one " Self supervision ".

The loss function consists of two parts , One is to restrict the image style to complete the conversion of the confrontation loss ：$${\mathcal{L}}_{\text{GAN}}(G,D_Y,X,Y)={\mathbb{E}}_{y\sim p_{\text{data}}(y)}[\log D_Y(y)]+{\mathbb{E}}_{x\sim p_{\text{data}}(x)}[\log (1-D_Y(G(x))]$$

This loss is necessary as long as style conversion is done , There's nothing to say . The other is the cyclic consistency loss of keeping the constraint content consistent ：$${\mathcal{L}}_{\text{cyc}}(G,F)={\mathbb{E}}_{x\sim p_{\text{data}}(x)}[\Vert F(G(x))-x{\Vert }_1]+{\mathbb{E}}_{y\sim p_{\text{data}}((y)}[\Vert G(F(y))-y{\Vert }_1]$$

For this kind of " Unsupervised " In terms of image style conversion , The upper limit of its effect is Pix2Pix such " Supervised " In the form of .CycleGAN One of the main problems of is the inability to deal with geometric transformations , Because the loss of cyclic consistency will make the content of the image as unchanged as possible in the process of converting to the target domain , That is, it is more likely to be " cat => cat => cat ", And it's hard " cat => Dog => cat ".