300+ documents! This article explains the latest progress of multimodal learning based on transformer

Address of thesis ：

https://arxiv.org/abs/2206.06488

Abstract

Transformer It is a promising neural network learner , It has achieved great success in various machine learning tasks . Due to the recent popularity of multimodal applications and big data , be based on Transformer Multimodal learning has become a hot topic in artificial intelligence research .

This paper deals with the problem of multi-modal data Transformer Technology has been thoroughly investigated . The main content of this article includes ：1） Multimodal learning 、Transformer Background of ecosystem and multi-modal big data era ;2） From a geometric topological perspective Vanilla Transformer、Vision Transformer and multimodal Transformer Theoretical review of ;3） Through two important paradigms , That is, multimodal pre training and specific multimodal tasks , For multimodality Transformer Review of application ;4） For multimodality Transformer Summary of common challenges and designs shared by models and Applications , as well as 5） Discussion of open issues and potential research directions in the community .