Can anime characters become "real people"? Paddegan helps you find the TA of "tear man"

This article is already in Flying propeller The official account is issued

Project background

With the generation of counter networks GAN Appearance , Interesting applications based on this technology have sprung up . One application direction , It is photo animation or video animation . Maybe it is influenced by the secondary meta culture , Or animation application scenarios are relatively extensive 、 High commercial value , As a result, many developers have participated in the production of the above photo animation type projects . After seeing too many similar projects , A question suddenly came to my mind —— Why is no one doing the opposite , Try to animate the project ？ meanwhile , I am also very curious to use GAN Generate anime characters “ Real people ” What will the image look like . therefore , Some time ago , I use Flying propeller PaddleGan Development Kit , It has realized a project of animation character live action , Give Way “ Tear up your girlfriend ” appear . Today, I want to share with you my project design ideas .

Project introduction

Technical principle

X: Original picture G(X): Generate graph G: generator D: Judging device Y: label

Simply speaking , The basic principle of this project , Is the original picture X And labels Y Send in GAN Online learning , The original image is generated by G Generate a human image G(X), Then generate the image G(X) And the original X Send to the discriminator D Judge , Then with the label image Y Calculation L1 loss and CGAN loss Modify the weight of the back propagation network , Make the final generated image G(X) can “ cheat ” Over discriminator D（ Mistakenly thought it was a label image Y）.

Implementation steps

1. Data preparation and preprocessing

• Data categories ： Animation photos （ Original picture ）/ Corresponding real person pictures （ label ）.

• Number of data ： About 1400 Zhang image .

• Data partitioning ： Training set 、 The proportion of the test set 13:1.

Corresponding real person pictures （ label ）/ Animation photos （ Original picture ）

2. Custom generator and discriminator

The generator uses Unet structure , Enter the target contour , After encoding and decoding, a colored target image is generated , The discriminator uses PatchGAN Distinguish the generated false image from the real image , Divide the image into slices patch, Judge each patch True or false , Then average the final result .

generator = UnetGenerator()discriminator = NLayerDiscriminator()out = generator(paddle.ones([1, 3, 256, 256]))print(' Generator output size ：', out.shape)out = discriminator(paddle.ones([1, 6, 256, 256]))print(' Discriminator output size ：', out.shape) The output is as follows ： Generator output size ： [1, 3, 256, 256] Discriminator output size ： [1, 1, 30, 30]

 Instantiate the generator 、 Judging device 

3. Define super parameters for training

#  Hyperparameters LR = 1e-4BATCH_SIZE = 8EPOCHS = 100#  Optimizer optimizerG = paddle.optimizer.Adam(    learning_rate=LR,    parameters=generator.parameters(),    beta1=0.5,    beta2=0.999)optimizerD = paddle.optimizer.Adam(    learning_rate=LR,    parameters=discriminator.parameters(),     beta1=0.5,    beta2=0.999)#  Loss function bce_loss = nn.BCELoss()l1_loss = nn.L1Loss()

We can check the effect of learning in the training process , As shown in the figure below .

Left ： in ： Right = Original picture ： label ： Generate the image

We can see it clearly ,pix2pix The effect of online learning is still very good , In particular, the image generated in the first line of the right image is very similar to the label image . Next, we use the test set to test the effect of the model .

4. test

#  Load weights for the generator weights_save_path = 'work/weights'last_weights_path = os.path.join(weights_save_path, sorted(os.listdir(weights_save_path))[-1])print(' Load weights :', last_weights_path)model_state_dict = paddle.load(last_weights_path)generator.load_dict(model_state_dict)generator.eval()#  Reading data # test_names = os.listdir('data/cartoon_A2B/test')# # img_name = np.random.choice(test_names)img_name = '01481.png'img_A2B = cv2.imread('data/cartoon_A2B/test/'+img_name)img_A = img_A2B[:, 256:]                                  #  Cartoon （ The input ）img_B = img_A2B[:, :256]                                  #  Human figure （ That is, the predicted results ）

We send cartoon character pictures to the network for prediction , For example, the following two pictures ：

You can imagine , Two cartoon characters in the picture “ real ” What will it look like ？

Project effect

Visualization of test results

Left ： in ： Right = Test chart ： Generate the image ： Test chart label

We can see , The effect of test chart generation is very close to the label image , Although some small details are not fully presented , but “ Real people ” The general appearance of the portrait can still be well reproduced . I don't know what you just imagined “ Real people ” How different is the appearance from the test effect ？

Problems and improvement direction

Problems to be optimized

• The style of data set is relatively simple （ The features are simple ）, Use photos of different animation styles to generate images , It may lead to poor generation effect . This is because there is still room for optimization of model generalization performance . in addition , At present, the data set used in this project is only female , Unable to generate images for men .

• The shadow part of the original image corresponds to the generated part, and the effect is generally poor . Besides , There is also the problem that the light sensitivity is not strong .

Improving direction

• You can try to optimize the dataset , That is, add the style type of data set 、 Gender , And maintain a certain amount of increase .

• Better generator models can be replaced , To strengthen the network to learn more detailed information .

Welcome to exchange

• Click to get the project link ：

  https://aistudio.baidu.com/aistudio/projectdetail/3450122

• More interesting experiences welcome to visit ：

  https://github.com/PaddlePaddle/PaddleGAN

This article pictures from ：
https://aistudio.baidu.com/aistudio/projectdetail/1813349?channelType=0&channel=0 Open dataset

Focus on 【 Flying propeller PaddlePaddle】 official account
Get more technical content ~

This article is shared in Blog “ Flying propeller PaddlePaddle”（CSDN）.
If there is any infringement , Please contact the [email protected] Delete .
Participation of this paper “OSC Source creation plan ”, You are welcome to join us , share .