"Multimodal" concept

Modality && Multimodal

Modality

Each source or form of information , You can call it a mode . for example , People have a sense of touch , auditory , Vision , The sense of smell ; Media of information , Have a voice 、 video 、 Words etc. ; A variety of sensors , Such as radar 、 infrared 、 Accelerometer, etc. . Each of the above can be called a mode .
meanwhile , Modes can also be very broadly defined , For example, we can think of two different languages as two modes , Even the data sets collected in two different cases , You can think of it as two modes .

Multimodal

therefore , Multimodal machine learning , English full name MultiModal Machine Learning (MMML), The aim is to achieve the ability to process and understand multi-source modal information by means of machine learning . At present, the hot research direction is image 、 video 、 Audio 、 Multi-modal learning between semantics .
Multi-mode learning from 1970 S start , It went through several stages of development , stay 2010 After full entry Deep Learning Stage .
A person is actually a sum of multi-modal learning , So there is also a ” Brick house “ Said the , Multi-mode learning is the real direction of artificial intelligence development .

from Multimodal definition