2022 T2i text generated image Chinese Journal Paper quick view-1 (ecagan: text generated image method based on channel attention mechanism +cae-gan: text generated image technology based on transforme

One 、ECAGAN: Text image generation method based on channel attention mechanism

Source of the article ： Computer engineering 2022 year 4 month
Citation format ： Zhang Yunfan , Yi Yaohua , Tang Ziwei , Wang Xinyu . Text image generation method based on channel attention mechanism [J]. Computer engineering ,2022,48(04):206-212+222.DOI:10.19678/j.issn.1000-3428.0062998.

1.1、 Major innovations

In the task of generating images for text The details of the generated image are missing And There is a structural error in the image generated in the low resolution stage （ If a bird has two heads , Lack of claws ） The problem of , Generate confrontation network based on dynamic attention mechanism （DMGAN）, Introduce content aware upsampling module and channel attention convolution module , A new method of text image generation is proposed ECAGAN.
The main innovations are ：

1. Adopt content aware up sampling method , The reconstructed convolution kernel is obtained by calculating the input characteristic graph , The convolution operation is carried out by using the reconstructed convolution kernel and the characteristic graph , Ensure semantic alignment ;
2. Use the channel attention mechanism to learn the importance of each feature channel of the feature map , Highlight important feature channels , Suppress invalid information , Enrich the details of the generated image ;
3. Combined with condition enhancement and perceptual loss function auxiliary training , Enhance the robustness of the training process .

1.2、 Main framework

The main structure is still similar StackGAN++ Three layer stacking , The network structure can be divided into Low resolution image generation stage and image refining stage , Generator generation in low resolution image generation stage 64×64 Pixel low resolution image , Generator generation in image refining stage 128×128 Pixels and 256×256 Pixel image .

1.2.1、 Low resolution image generation stage

Text encoder stage and StackGAN、AttnGAN、DMGAN And so on , Generate sentence features and word features through text encoder , The sentence features are stitched together after random noise FC Input to Content aware upsampling module （CAUPBlock） in , The first stage image is formed after up sampling the input feature map .

Content aware upsampling module It consists of an adaptive convolution kernel prediction module and a content aware feature reorganization module ：

Adaptive convolution In the nuclear prediction module , The feature map passes through the content encoder ,Reshape And normalized to a size of $SH\times SW\times k_{up}^2$ Recombined convolution kernel of . Content aware feature reorganization module Simply put, it is to dot product each region of the feature map with the corresponding predicted convolution kernel .

Structure diagram is as follows , Characteristics of figure R After input, in the adaptive convolution kernel prediction module ψ in
For output characteristic graph R′ Every area of l′ Predict the convolution kernel ${\gamma }_l$, Then the original feature map is in the content aware feature reorganization module ξ Neutralize the predicted convolution kernel and dot multiply to get the result $\begin{array}{l}{\gamma }_{l^{\prime }}=\psi \left(Z\left(R_l,k_{\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}}\right)\right)\\ R_{l^{\prime }}^{\prime }=\xi \left(Z\left(R_l,k_{\mathrm{u}\mathrm{p}}\right),{\gamma }_{l^{\prime }}\right)\end{array}$, among $Z（R_l,k_{up}）$ Represents the midpoint of the feature map l The surrounding $k_{up}\times k_{up}$ Sub areas of .

After up sampling, the characteristic graph is input into the generator , After convolution operation with channel attention mechanism, the image is generated .
The attention convolution module weights the feature map through channel attention , Make the generated image more detailed , The realization of channel attention will not be repeated , See the original text for details .

1.2.2、 Image refining stage

Image refining stage and AttnGAN Very similar , The dynamic attention layer is used to calculate the correlation between each word in the word vector and the image sub region , Then calculate the attention weight of the image sub region according to the Correlation , Finally, the update of the feature map is controlled according to the attention weight of the feature map , Get a new feature map and then upsampling , Zoom in on the image .（ Three methods that can effectively fuse text and image information second ）

1.3、 Loss function

The generator loss function is in the form of ：

$L_G=\sum \left(L_{G_i}+{\lambda }_1{L}_{\mathrm{p}\mathrm{e}\mathrm{r}}\left(I_i^{\prime },I_i\right)\right)+{\lambda }_2{L}_{\mathrm{C}\mathrm{A}}+{\lambda }_3{L}_{\mathrm{D}\mathrm{A}\mathrm{M}\mathrm{S}\mathrm{M}}$

$L_{Gi}$ Represents the loss function of generators at all levels ;$L_{per}$ Represents the perceived loss function ;$L_{CA}$ Represents the conditional enhancement loss function ;$L_{DAMSM}$ Express DAMSM Module loss function .

$L_{G_i}=\underset{\text{unconditional loss }}{\underbrace{-\frac{1}{2}{E}_{x\sim p_{ϵ}}\left[{\log }_a{D}_i\left(\widehat{x_i}\right)\right]}}\underset{\text{conditional loss }}{\underbrace{-\frac{1}{2}{E}_{x\sim p_{\sigma },}\left[{\log }_a{D}_i\left({\hat{x}}_i,s\right)\right]}}$

$\begin{array}{l}{L}_{D_i}=\\ \underset{\text{unconditional loss }}{\underbrace{-\frac{1}{2}{E}_{x\sim p_{\text{datu }}}{\log }_a{D}_i\left(x_i\right)-\frac{1}{2}{E}_{x\sim p_{{\sigma }_i}}{\log }_a\left(1-D_i\left(\widehat{x_i}\right)\right)}}\\ \underset{\text{conditional loss }}{\underbrace{-\frac{1}{2}{E}_{x\sim p_{\text{datu }}}{\log }_a{D}_i\left(x_i,s\right)-\frac{1}{2}{E}_{x\sim p_{{\sigma }_i}}{\log }_a\left(1-D_i\left({\hat{x}}_i,s\right)\right)}}\end{array}$

$\begin{array}{l}{L}_{\mathrm{C}\mathrm{A}}=D_{\mathrm{K}\mathrm{L}}(\mathcal{N}(\mathbf{\mu }(\mathbf{s}),\mathbf{\Sigma }(\mathbf{s}))\Vert \mathcal{N}(0,I))\end{array}$

$\begin{array}{l}{L}_{\mathrm{p}\mathrm{e}\mathrm{r}}\left(I^{\prime },I\right)=\frac{1}{C_i{H}_i{W}_i}{\Vert {\varphi }_i\left(I^{\prime }\right)-{\varphi }_i(I)\Vert }_2^2\end{array}$

1.4、 experiment

Two 、CAE-GAN: be based on Transformer Cross attention text generation image technology

Source of the article ： Computer science 2022 year 2 month
Citation format ： Tan Xinyue , He Xiaohai , Wang Zhengyong , Luo Xiaodong , Sparkling waves . be based on Transformer Cross attention text generation image technology [J]. Computer science ,2022,49(02):107-115.

2.1、 Major innovations

at present , The mainstream method is to complete the encoding of the input text description by pre training the text encoder , but Current methods encode text descriptions , The mapping relationship with the corresponding image is not considered , Ignoring the semantic gap between language space and image space , As a result, the matching degree between the generated image and the text semantics in the initial stage is still low , And the image quality is also affected .

Innovation points ：

Through the cross attention encoder , Translate and align text information with visual information , To capture the cross modal mapping relationship between text and image information , So as to improve the fidelity of the generated image and the matching degree with the input text description .

2.2、 Main framework

Pictured above , The text first passes through the cross attention encoder , Generate cross attention eigenvectors $f_c$ And word feature matrix W, The cross attention feature vector is fully connected after adding noise 、 Initial image features are formed in the upper sampling stage , Then word characteristic matrix Ｗ Input to dynamic memory module and fuse with primary image features , Get the new image features after fusion , The process is as follows ：

$\begin{array}{l}{\mathbf{f}}_c,\mathbf{W}=C_E\left(\mathbf{s},{\mathbf{F}}_R\right)\\ {\mathbf{F}}_0=G_0\left({\mathbf{f}}_c+\mathbf{z}\right)\\ {\mathbf{F}}_1=G_1\left(DM\left({\mathbf{F}}_0,\mathbf{W}\right)\right)\\ {\mathbf{F}}_2=G_2\left(DM\left({\mathbf{F}}_1,\mathbf{W}\right)\right)\end{array}$

2.2.1、 Cross attention coder

Cross attention coder is used for joint cross coding and alignment of language information and visual information . As shown in the figure below ：

It mainly includes text feature extraction 、 Image feature extraction 、 Cross attention coding and self attention coding ：

1. Text feature extraction , Use two-way LSTM Encode the original text into a global sentence feature vector W And a word eigenvector s;

2. Image feature extraction , Use InceptionV3 The Internet Extraction of image features $f_c$;

3. Cross attention code , It is mainly used to construct the internal relationship between language features and image features , Realize joint coding . Word eigenvector s And image features $f_c$ They are mapped into $q_s,k_s,v_s,q_v,k_v,v_v$, namely $q_s,k_s,v_s=Linear(s),q_v,k_v,v_v=(f_v)$, Then calculate the cross attention score score, Then get the attention code $l_c$
   $\begin{array}{l}\text{ score }={\lambda }_c{\mathbf{q}}_v{\mathbf{k}}_s^{\mathrm{T}}\\ \text{ score }=\mathrm{Softm}(\text{ score })\\ {\mathbf{s}}_c=\mathrm{dropout}\left(\text{ score }{}^{\prime }\right)\\ \mathbf{l}={\mathbf{s}}_c\cdot {\mathbf{v}}_s,l_c=Normalization(A_{1}l+B_1)\end{array}$

4. Self attention encoding , The main use is still Self attention mechanism babbling , I won't repeat
   $\begin{array}{l}{\mathbf{q}}_l,{\mathbf{k}}_l,{\mathbf{v}}_l=\text{ Linear }\left({\mathbf{l}}_c\right)\\ {\mathbf{s}}_s=\mathrm{Dropout}\left(\mathrm{softm}\left({\lambda }_s{\mathbf{q}}_l{\mathbf{k}}_l^{\mathrm{T}}\right)\right)\\ {\mathbf{l}}_{cs}={\mathbf{s}}_s\cdot {\mathbf{v}}_l\\ {\mathbf{f}}_c=\mathrm{Normalization}\left(A_2{\mathbf{l}}_{cs}+B_2\right)\end{array}$

2.2.2、 Dynamic enclosure

Dynamic enclosure and DM-GAN The internal dynamic memory mechanism is similar .

2.3、 Loss function

The global loss function L Divided into three parts , The total loss function is as follows ：

$L={\sum }_i{L}_{G^i}+{\tau }_1{L}_{CA}+{\tau }_2{L}_{DAMSM}$

among $L_{G^i}$ Is the generator loss ,$L_{CA}$ Is the conditional loss function ,$L_{DAMSM}$ It is the loss of deep attention multimodal similarity （ And AttnGAN be similar ）

$\begin{array}{rl}{L}_{G^i}& =-\frac{1}{2}\left[E_{x\sim p{G}^i}\log D_i(x)+E_{x\sim p{G}^i}\log D_i(x,s)\right]\\ L_{CA}& =D_{KL}(N(\mu (s))\Vert N(0,I))\end{array}$

The loss function of the discriminator consists of conditional loss and unconditional loss ：

$\begin{array}{l}{L}_{D^i}=-\frac{1}{2}\left[L_D+L_{CD}\right]\\ L_D=E_{x\sim P\text{ data }}\log D_i(x)+E_{x\sim p{G}^i}\log \left(1-D_i(x)\right)\\ L_{CD}=E_{x\sim P\text{ data }}\log D_i(x,s)+E_{x\sim p{G}^i}\log \left(1-D_i(x,s)\right)\end{array}$

2.4、 experiment

Last

Personal profile ： Graduate students in the field of artificial intelligence , At present, I mainly focus on text generation and image generation （text to image） Direction

Personal home page ： Medium coke with more ice

Time limited free subscribe ： Text generated images T2I special column

Stand by me ： give the thumbs-up + Collection ️+ Leaving a message.

in addition , We have established wechat T2I Learning exchange group , If you are T2I Fans or researchers in this field can send me a private message to join .

If this article helps you a lot , I hope you can give me a coke ！ Add more ice ！