I'm "fixing movies" in ByteDance

“ What supports us to complete such a complicated work ？ Because we understand their value to future generations —— The so-called classic video , It's a long-standing content , They deserve to be revitalized with the help of new technology .”

When 4K Fixed version of 《 Legend of Nezha 》《 Brother Hulu 》《 Sheriff black cat 》《 Big head son and small head father 》《 The adventures of little carp 》《 Tadpole looking for his mother 》6 Watermelon video of a classic cartoon online App after , Zhen Chen, the product manager, sighed .

Six old animations 4K The repaired version has just been online for two weeks , More than a million hits . And half a year ago , When they first started the work, they were far from expecting , An old animation that you think you can achieve with a super-resolution algorithm 4K Repair project , It's so repeated and difficult ：

• The old film is badly damaged and scratched ;
• The limitations brought by art style are huge ;
• vision / Close range is very different , Fine texture is difficult to repair directly , It is difficult to maintain the original art style ;
• ……

Behind these clear images , What has happened to the R & D team of byte beating volcano engine ？

The road of repairing pits frequently

4K Most of the technical students of the old animation restoration project are 90 after , About half of them are new recruits who have just joined the school this year , Repairing these old animations is also repairing their childhood memories .

Ge Ling of the engineering team of the project team said , When I was a child, the first cartoon CD my parents bought for me was huluwa , The film has been with him for a long time , Now start again 《 Brother Hulu 》 The restoration of , My heart is full of childhood memories .

Memories are beautiful , The real repair work is very difficult .

The situation of each cartoon is different , Some are kept on film , There are many damages caused by scratches and adhesion ; Some are saved in digital format , But the sawtooth of the enlarged line is very serious ; Some lines are clear 、 The color is highly saturated , Others use artistic styles such as ink painting ,「 Fix movies 」 The traditional way , It can't be perfectly applied to these cartoons with different art styles .

so to speak , Every film is different , We can't rely solely on super-resolution algorithm , Instead, each algorithm engineer is responsible for 2~3 Animation department , According to its characteristics, the customized technical scheme is designed separately .

Fu Ren, an algorithm engineer, just graduated from Shanghai Jiaotong University with a master's degree when he entered the project , Through campus recruitment, I came to Beijing .

The cartoon she participated in is 《 Brother Hulu 》 and 《 Sheriff black cat 》. Both of them are film films , The first step to getting the film , She completed a multi-dimensional quality analysis for two cartoons , Investigated many different methods of deep learning and traditional learning , Developed a scratch removal module .

But after trying to run, I found , The scratches and noise on the old film are too serious , She thought of using deep learning to complete the scratch detection , But I didn't think , After adding the scratch detection module , The result is not as good as expected ：

The accuracy of the algorithm cannot reach 100%, It is possible to treat the texture of a small part of the pattern in the animation as a scratch , Erase them together .

The leaves on the left are removed as film stains

Butterfly wings in the air , Easily confused with film stains

Although animation data is used to train the scratch detection model , But this problem is still unsolvable , After discussion , In order to ensure the perfection of each repaired picture , The team finally decided 「 Algorithm + artificial + Algorithm 」 A three-step approach ：

• First use the algorithm to enlarge the video to 4K, Preliminary noise removal , Complete picture enhancement , Make the cartoon clearer ;
• Then find the marked supplier , Find those scratches that look very obvious to people's eyes , For example, the part scratched by the projector 、 The adhesive part of film in years of storage , Based on this, draw a mask frame by frame ;
• Then with the help of repair algorithm , Fill up these manually marked damage parts .

This is a set of manual 、 Also in an intelligent way , After the second step of manual annotation, the third step of algorithm filling , The result of the machine running out is often not satisfactory ,“ Maybe the stroke of the marking party accidentally crossed the boundary , The color block overflows its original position ; The blurred area on the picture , After improper marking , The color patch will be abnormal , The human eye looks very abrupt ,” Project team's PM Classmate Zhen Chen said , He has to set aside a certain amount of time every day , For these imperfect running results 「 Batch work 」, After repeated modification for several times , He has been to 《 Brother Hulu 》 and 《 Sheriff black cat 》 The plot is familiar to my heart ,“ Episode 1 10 Minutes of animation , I have to watch it for an hour to find out all the imperfect details , Over and over again , Finally produced the perfect version .”

The problem with these texture details , It also appears in 《 The adventures of little carp 》 On .

Responsible for completing the restoration of this cartoon hasaki It is also the enrollment of the school that joined this year , Graduated from Xi'an University of Electronic Science and technology . In repeated experiments ,hasaki Found a strange frame ：

When using an algorithm to repair , Whenever you encounter a small carp swimming in the water , The processed subtitles occasionally appear distorted .

such as , Normally, it should be like this ：

Will twist like this , The outline of the text is blurred out , The stroke thickness is also random , Even part of the text becomes very abstract ：

It was found by experiment that , On the one hand, the texture of subtitles has the characteristics of large gradient and dense , On the other hand, the caption itself is still , But the background is always moving , There will be interference . Under the influence of these two factors , After an algorithm is repaired, it is easy to produce badcase.

In response to this situation ,hasaki Modified his plan , The subtitle area is recognized , A separate optimization strategy is adopted for the caption area , Avoid this problem .

In addition to algorithm students , The engineering team has also made many improvements .

Base Ge Ling in Hangzhou began to participate in this project after graduating from Zhejiang University ：“ I almost just graduated 、 Research on video enhancement technology began in July and August ,9 I joined the project team in June .”

Different from the algorithm side , Realize the of old animation 4K Repair , The engineering side needs to make these different algorithm processing methods run in a general 「 batch production 」 In the process , Create a pipelined processing flow .

However , These different ages 、 Different ways of making old animation have different formats , Want to output as 4K The resolution of the 、 16：9 The form of , The details and sequence of processing need to be considered based on various film sources , It's a difficult engineering challenge . Ge Ling said ：“ Some sources even maintain the pixel ratio of the TV system , Each pixel is rectangular , Unlike computers 、 Modern system 1:1 Square .”

Because of the complexity of this source , How to uniformly deploy the personalized algorithm module of each old animation on the mass production pipeline , Ensure the normal operation of different modules , Also make sure they are compatible with each other , It's a process that needs constant polishing , Every time you find a new problem , We have to reproduce the problem 、 Location, troubleshooting and repair .

This complex system also requires its maker's ability to control the details ,“ An unimportant parameter that is not aligned may lead one hair and affect the whole body ”, Ge Ling said . For each source , All need the concerted efforts of engineering students and algorithm students , After customizing the workflow, the process deduction is carried out in combination with the engineering code , Sort out the details and pipeline in specific video processing .

See the release from the internal trial , The classic finally reappears

After many rounds of modification and adjustment , When all the old animations have been repaired , The time to meet the challenge has finally opened .

This is a viewing meeting before the official launch , Will determine whether the repaired film can be officially launched .

The organization of the film viewing meeting can be said to have a sense of ceremony , All the students who participated in the project came to a family in Beijing 4K In the cinema , Play with professional equipment , Review 4K The effect of repair .

Before that , Fu Ren in Shanghai and... In Shenzhen hasaki Also came to Beijing on business , Prepare for the film viewing three days in advance .

I've never been in touch with what the cinema plays hasaki I learned a lot of new knowledge during this period ：“ I didn't think of it , When movies are shown in the cinema , Even the format is different from that of our ordinary users , It's not a regular format .”

original , The movie played in the cinema is equivalent to cutting the video of our ordinary users into frame by frame , Save it in this framing format , Then modify the bit rate . Besides , Considering the screen size of the theater , It is also often necessary to add black boxes around the movie screen to maintain the fit .

Just came into contact with this system hasaki Quickly learned the transcoding tool for cinema films , And produce multiple demonstrations within three days demo For internal viewing .

In the process of preparing for the internal film viewing meeting , They also tried to play it many times on the large screen of the cinema . The playing effect of large screen is very different from that of computer screen , Zoom in a few times , Some unclear details are magnified again , It needs constant adjustment and optimization to achieve the best playback effect .

After hard preparation , When all project students come to the scene , Clear 、 The beautiful playing effect allows everyone on the scene to revisit their childhood memories with clear picture quality , Everyone agreed to put these repaired films online .

later , In the Chinese film director Center ,300 More than watermelon video users 、 Media friends and partners witnessed these old animations appear in front of the world with a new look at the press conference .

And the R & D team behind these young students' continuous efforts , Finally seen by the world . Their pursuit of excellence and the pursuit of extreme details , It has become a part of the classic , Stay in the memory of the audience .

Improve yourself , Transmission value

“ original 《 Sheriff black cat 》 Only five episodes , I feel like when I saw it when I was a child , It seems that there are many, many episodes .” After project completion , Fu Ren can finally look at these old cartoons with ease .

After assuming the role of some work leaders in some projects , She felt herself growing up quickly ：“ my mentor Will consciously let me be responsible for some things , I will feel that I am doing the project management of my own work , Communicate and cooperate with different students , Run through the process of the whole project .”

In fact, this is the feeling of major enrollment after entering this project team ： At school , The rule of doing things is what the tutor arranges , What do you accomplish by yourself , But now you need to take the initiative to arrange what you want to accomplish , Do your own experiments , I take the initiative to prove to other partners that my version will be better than another version .

And in the method of technical work , The students in the project team also have a lot of insights , People gradually realized that , Many problems in film repair need to be considered from multiple angles from the root , Then consider some improvements in the algorithm .

Goling on the side of the project is also the first time to participate in such a large project , The implementation of this whole process has become the first highlight of his career ：“ The pipeline method of this project is what I thought of , Finally applied it to the project , It gives me a great sense of achievement .” He is mastering a mature way of doing things , To put their skills on the ground .

Volcano engine multimedia lab is responsible for byte beat video enhancement , Faced with very rich and difficult scenes , It can be said that it is the place with the richest video enhancement business scenes in China . Laboratory researcher Zhao Shijie thinks , Old animation is really hard to repair , Each student is required to propose technical solutions for complex scenes , Not only to ensure that the image quality reaches 4K standard , And the original artistic beauty must also be inherited , This is also the embodiment of each student's pursuit of perfection .

They are continuing the work of repairing old animation , Watermelon video will also open the entrance , Free for ordinary users AI Repair support .

Now? , If you open watermelon video App, You can see this 6 Ministry 4K HD cartoon . They are everyone's image memories , From the early classics on the screen to works that can be viewed at any time on the mobile terminal , It is becoming a resonance across generations .

Repair these common memories with solid technology , Serve every audience that has been moved by it , It is the value pursued by pragmatic technicians .