Multi modal data can also be Mae? Berkeley & Google proposed m3ae to conduct Mae on image and text data! The optimal masking rate can reach 75%, significantly higher than 15% of Bert

This article shares papers 『Multimodal Masked Autoencoders Learn Transferable Representations』, reflection ： Multimodal data can also be MAE？UC Berkeley&Google Put forward M3AE, On image and text data MAE！ The optimal masking rate of text can reach 75%, Significantly higher than BERT Of 15%！

The details are as follows ：

• Thesis link ：https://arxiv.org/abs/2205.14204

Build extensible models to diversify 、 Learning from multimodal data is still an open challenge . For visual language data , The main method is based on comparative learning objectives , That is, a separate encoder is trained for each mode . Although effective , However, the comparative learning method will introduce sampling bias according to the data used , This reduces the performance of downstream tasks . Besides , These methods are limited to paired image text data , Inability to leverage widely available unpaired data .

 

In this paper , The author studies a scheme that only uses masking token A large multimodal model for predictive training , Without using a modal specific encoder or comparative learning , You can learn the transferable representation of downstream tasks . The author proposes a simple and scalable network architecture , That is, multimode masking automatic encoder （Multimodal Masked Autoencoder,M3AE）, It's through masking token A unified coder for predictive learning of visual and linguistic data .

 

The author is interested in training on large-scale image text data sets M3AE Conducted an empirical study , Find out M3AE Be able to learn transferable representations , And can be well migrated to downstream tasks . Due to the joint training of the two data modes , And the standard masking rate is 15% Of BERT comparison ,M3AE Benefit from higher text masking rate （50-90%）. The author also provides qualitative analysis , It shows that the learned representation combines meaningful information from image and language . Last , The author shows M3AE extensibility , It has a larger model size and training time , And the flexibility of training on paired image text data and unpaired data .