The five minute demonstration "teaches" actors to speak foreign languages and can seamlessly switch languages. This AI dubbing company has just received a round a financing of 20million US dollars

Line early From the Aofei temple
qubits | official account QbitAI

It only takes five minutes of actors' voice material , You can make him speak another language in the movie ？

I don't believe it until I see this video , Listen to the effect of this passage ：

This video is taken from 《 Son of Bodo 》（ English name Every Time I Die）, Is a english Thriller .

But we can see in the broadcast , Just one click , You can convert English into Spanish at any time , And it still sounds like the voice of the original actor .

Even talking in horror 、 The trembling details were faithfully inherited , Show us a AI The magical power of dubbing .

Of course , This wave of operation has not surprisingly moved many investors .

The company that made this paragraph Deepdub （ Deep dubbing ）, Recently in A Got... In the round of financing 2000 Thousands of dollars . Among the investors are the former president of Fox TV studio 、Snyk Co-founder of 、Meta Vice president of engineering, etc .

AI Dubbing impacts the traditional mode

AI Why is dubbing so expected ？ Because it contains huge business opportunities .

Need to know , English audiences in the United States and other places are not used to watching subtitles . therefore , Facing some excellent works in non English , They have a strong Localization needs , That is, the English dubbing version .

For example, some time ago, the fire broke out Korean dramas 《 Squid game 》, At the premiere 28 Days. , The total viewing time is 16.5 100 million hours , Add up to 18.2 In ten thousand, . Become at one stroke Netflix The number one program in history .

But such a big cake , From a traditional point of view , It's very hard to eat .

△ Figure note ：《 Squid game 》 Play volume , The first row in the right column

for example , Local publishers have to spend money translating the script , We have to hire a voice actor to play the role 、 Rent space and equipment 、 Complete a lot of dubbing and recording , Finally, we need to splice the dubbing into the original video .

There are also many cultural differences .

This one comes down , According to the market, we should 15-20 Zhou .

and Deepdub Of AI The dubbing method only requires the original actor to record five minutes of random text , Let the neural network learn the actor's voice and express it in another language .

It sounds like the original actor learned another language , And the same workload can be completed in only four weeks , Including translation 、 Adaptation 、 Mixing, etc .

In terms of technical details ,Deepdub Not much public , Maybe it can be used in GitHub On fire Mocking Bird Make reference .

It only takes five seconds , You can clone any Chinese voice , Then use the same voice color to synthesize other voice content , Realize the process from voice to text and then to voice .

The model structure is mainly composed of the speaker encoder （Speaker encoder）、 Synthesizer （Synthesizer） Harmony coder （Vocoder） form .

The speaker encoder （ green ） Extract the feature vector of the speaker's voice , Learn timbre .

Then the traditional TTS（Text-to-Speech） link ：

In the synthesizer （ Blue ） The speech features are integrated into the specified text , Take the Mel spectrum as the intermediate variable , Transmit the generated speech spectrum to the vocoder （ Red ）.

Finally, the depth autoregressive model WaveNet As a vocoder , Use the spectrum to generate the final speech .

however ,Deepdub Although he didn't disclose his technical details , But they claim to have taken the lead in this field of academic research .

This is also a bit credible , From their products 、 The investment obtained and the background of brother founders can also be seen ：

Younger brother Nir Krakowski Yes 25 Years of professional R & D experience , brother Ofir Krakowski He also worked in the machine learning Department of the Israeli air force ……

AI There are many racing cars in dubbing track

Of course , There are more than just people who like this market Deepdub a , It's just a little different in strategy .

Deepdub It is the way to modify the audio , The video content remains intact . They plan to use this round of financing to expand the team's marketing 、 Research and engineering department , And is talking about cooperation with Hollywood .

British companies Papercup Methods adopted and Deepdub similar , Also focus on audio , Redeploy the original actor's voice through the flip , Use synthetic sound , Keep the video the same .

And the other one Flawless In audio, we also rely on dubbing actors , But I can edit the face and mouth shape in the video , It looks more like speaking the target language .

Like the others , Amazon and other technology giants are also doing relevant research , But there is no product yet .

So it seems , Maybe we can really create the video industry in the future “ Babel Tower ”, Make barrier free communication in online drama .

Or, , Some individual actors really don't have to memorize their lines ？

Reference link ：

[1]https://techcrunch.com/2022/02/10/deepdub-raises-20m-for-a-i-powered-dubbing-that-uses-actors-original-voices/
[2]https://venturebeat.com/2022/02/10/deepdub-closes-fresh-financing-round-for-ai-that-dubs-movies-shows-and-games/