Learning pyramid context encoder network for high quality image painting paper notes

IEEE Conference Proceedings arXiv: Computer Vision and Pattern Recognition Jan 2019

Problems solved and improvement

Existing methods cannot be combined Direct visual information and deep semantic information .

1. patch search And others lack the understanding of high-level semantic consistency .
2. generative models Of stacked constructions and poolings There is over-smooth, lack of visually-realistic Other questions .

Model structures,

With UNet For the skeleton , In the image-level and feature-level Fill the missing area on .

pyramid-context encoder： Use cross-layer The mechanism of attention transmission and pyramid filling

Each level 𝜓 From this layer feature map - 𝜙 and On a higher level 𝜓 Common process ATN（ In style f） obtain .

Attention Transfer Network(ATN)（ It's the one above f）

One 、 Reconstruct feature map from high-level semantics ${\psi }^L$ Fill in the next layer of feature map ${\varphi }^{L-1}$, To get the reconstruction feature map of the next layer ${\psi }^{L-1}$.

1. First extract ${\psi }^l$, And then calculate patch Cosine similarity between .
   

2. Then use on similarity Softmax Function to get each patch My attention score （Attention Score）.
   

3. After obtaining the attention score of high-level semantic features （ Namely the above formula ${\alpha }_{i,j}^l$）, The feature map of the next level can be weighted by the attention score context Fill in .

Calculate all patch after , You can get ${\psi }^{l-1}$ （ above i All calculations of can be formulated into convolution calculation for end-to-end training ）.

Two 、 elaboration

The multi-scale context information is aggregated by four groups of dilated convolutions with different rates , This design ensures the consistency between the structure of the final reconstruction feature and the environment , Improved the repair effect of the test .

multi-scale decoder

• multi-scale decoder Approved by ATN Reconstruction features and encoder Of latent feature Make input .
• decoder Characteristic graph ${\phi }^{L-1}、{\phi }^{L-2}$ etc. , It is calculated from the following formula .

among , from ATN The generated reconstruction feature is that the missing region encodes lower level information , It is beneficial to use fine-grained details to generate visually realistic results ; Compact extracted by convolution latent When the feature can't find the object in the area outside the missing , Synthesize new objects .
Semantic consistency depends on deep convolution , The texture is consistent ATN Shallow features of reconstruction .

• Pyramid L1 losses
  

An adversarial training loss

The total loss function consists of ：Generator + Discriminator

• Use PatchGAN（Image-to-Image Translation with Conditional Adversarial Networks） As part of this article discriminator, At the same time, spectral normalization is used to stabilize the training .
• In this paper ,pyramid-context encoder and multi-scale decoder constitute Generator.

The definition of the loss function ：

1. Definition generator The final prediction result z：
   $$z=G(x\odot (1-M),M)\odot M+x\odot (1-M)$$

2. discriminator The confrontation loss function of can be expressed as ：
   

3. generator The confrontation loss function of is ：
   

4. PEN-NET By minimizing counter losses and pyramid L1 Loss （ At the end of the last section ） To optimize , The overall objective function is ：
   

model analysis

analysis pyramid L1 Loss and ATN The role of these two network components .

Pyramid L1 Loss

Pyramid L1 Loss The loss function is gradually refined at each scale ,pyramid loss It is conducive to decoding compact features layer by layer .

ATN

Cross layer attention transmission mechanism to U-Net Skeleton brings improvement .

The first behavior is pure... Without using any attention mechanism U-Net The Internet , The second line is no deeper guidance Of CA Method , The third layer is ATN Apply to U-Net Architectural results .