Meta enlarge again! VR new model posted on CVPR oral: read and understand voice like a human

Meta This research mainly includes three models , They are visual and acoustic matching models （Visual Acoustic Matching model）、 Vision based de reverberation model （Visually-Informed Dereverberation）、 Audio video separation model （Visual Voice）.

First , The visual acoustic matching model can transform the audio in the video into the acoustic effect in the target environment , Given the image of the target environment and the waveform of the source audio , The model can then re synthesize the audio to match the acoustics of the target room .

 

Then there is our vision based audio de reverberation model (VIDA), It can learn to eliminate reverberation according to the observed sound and visual scene .

 

and Visual Voice The model can separate the audio and video in the video across the model .

Address of thesis ： https://arxiv.org/pdf/2202.06875.pdf