Sell notes | brief introduction to video text pre training

author ： Harbin Institute of technology SCIR Zhong Weihong

1.  brief introduction

With the development of the pre training model , Researchers also began to try to apply the framework and method of pre training model to multimodal tasks . In the picture - Text multimodal tasks , The application of pre training model has achieved excellent performance . Compared to pictures , The information contained in the video content is more abundant and redundant , Multiple frames may contain highly similar pictures . Different from the picture , Video content naturally contains timing information , As the length of video grows , It contains more and more time series information . meanwhile , Because the volume of video data is larger than that of pictures , Data sets 、 The construction of models has posed a greater challenge for researchers . therefore , How to be more elegant , Build video with high quality - The connection between textual representations 、 Have a good interaction , And bring improvement to downstream tasks , It has become a problem for researchers to explore .

This article briefly combs the current video - The model architecture of text pre training and related data sets , meanwhile , Aiming at the redundancy of video information , The work of introducing fine-grained information is briefly introduced .

2. Commonly used pre training data sets

The data of multimodal pre training usually comes from large-scale modal alignment sample pairs . Due to the existence of temporal dimension , Video contains more abundant and redundant information than pictures . therefore , Collect large-scale videos - It is difficult to use text alignment data for video pre training . at present , The public pre training datasets used by most researchers mainly include HowTo100M[1] and WebVid^[2] Data sets , Besides , Due to the similarity of video and picture features , There is also a lot of work using pictures - Text pre training data set for training , This section focuses on video - The commonly used data sets in text pre training are briefly introduced .

2.1 HowTo100M

Learning video - The cross modal representation of text usually requires manual annotation of video clips (clip), Labeling such a large data set is very expensive .Miech^[1] And so on HowTo100M Data sets , Help model from voiceover text with auto transcribe (automatically transcribed narrations) Learn the cross modal representation from the video data of .HowTo100M from 1.22M A teaching with narration (instructional) In the network video, you can get 136M A video clip (clip). The teaching content of video is mostly displayed by human beings , It contains more than 23, 000 different visual tasks .

chart 1 HowTo100M Data set Overview

Researchers from WikiHow To retrieve the 、 Extracted 23,611 A visual task that can interact with the physical world , And in YouTube Search for relevant videos by constructing keywords , Keep the video with English subtitles . English subtitles through the timeline and specific video Clip Make up the video - The text is right , An example is shown in the figure 2 Shown .

chart 2 Video in data set - Examples of text pairs

And the previous video - Text pairing data sets are different ,HowTo100M The descriptions in the dataset are not manually annotated , Because of the nature of narration , It corresponds to the corresponding video clip The timing may not be aligned , It may not be a complete sentence . But the data set is large enough , For video - The work of text pre training further provides the possibility .

2.2 WebVid

WebVid-2M ^[2] The data set contains the data crawled from the network 2.5M video - Text data pairs , And HowTo100M Different ,WebVid The video data contained in comes from the general field .WebVid Data set construction and CC3M ^[3] similar , Researchers found that CC3M In more than 10% In fact, all of the pictures are the overview of the video , By finding the original video , Researchers can crawl 2.5M In the video - The text is right .

chart 3 Shows some examples from the dataset ,WebVid There are various styles of video description in , Contain concise or detailed descriptions .

chart 3 WebVid An example of a dataset

surface 1 The statistics of some existing video text data sets are listed , Compared with HowTo100M Data sets ,WebVid Is less than its size 1/10. But the text in the dataset is usually a manual description , Have a good sentence structure , Better match with video , At the same time, it also avoids ASR Grammar errors caused by writing .

surface 1 video - Statistics for text datasets ^[2]

2.3 Conceptual Captions

Video is composed of multiple frames of still pictures , It is difficult to collect video resources , Researchers also often use large-scale image text data sets to expand the scale of pre training data , Enhance the generalization of the model .Conceptual Captions ^[3] from Google Researchers at 2018 Released in , Researchers extract from billions of web pages 、 Filtering results in large-scale images - Describe text pairs . To make the picture description cleaner and easier to use , It automatically generates the corresponding transformation description based on the description text and picture , be called Conceptual Captions. The statistical data of the data set is shown in table 2 Shown .

surface 3 Conceptual Captions Statistical data ^[3]

3. Model architecture

video - The text pre training model involves video 、 Text encoding and processing , Generally, it can be roughly divided into single streams (Single-Stream) Architecture and multi stream (Multi-Stream) framework . about Single-Stream Model of Architecture , Characteristics of different modes / Indicates that it is input into a separate cross modal encoder , Capture their modes inside / Interactive information between . And for Multi-Stream framework , video 、 The text will be input into the respective modal independent encoders , Capture the representation information inside the mode , Then establish the cross modal connection through different ways . This section will briefly introduce the models of different architectures , For models that introduce more granular information , We will introduce it in the following chapters .

3.1 Single-Stream

3.1.1 VideoBERT

VideoBERT ^[4] Is the first to use Transformer Architecture exploration video - Pre training model of text representation . from BERT ^[5] Inspired by , Researchers try to quantify video content , For video Clip Encoding , clustering , The discrete representation is extracted from it , Call it visual words (video words), For every video , They can be expressed by multiple visual words , And it can be input into the encoder together with the text for joint representation learning , Its architecture is shown in the figure 4 Shown .

chart 4 VideoBERT Model architecture ^[4]

And BERT be similar ,VideoBERT Introduced text 、 Two modes of video mask completion task (mask-completion), To be specific , It is to use context and cross modal information to recover [MASK] Tag covered video / Text token. meanwhile , In order to model the correlation between different modes ,VideoBERT Also use [CLS] The encoding of tag position determines whether the sequence between video and text / Semantic alignment .

3.1.2 ClipBERT

Previous work usually uses the dense data extracted by video coders pre trained in different fields (dense) Video features ,ClipBERT ^[6] It takes advantage of the similarity between video and pictures , By thinning the video (sparse) It is encoded by sampling , And realize the end-to-end pre training .

To be specific , Researchers randomly took multiple video clips (Clip), Sparse sampling for each video segment , The video frame is encoded to get the representation ; For different Clip Representation of different frames , Information fusion in time and space , The resulting representation will be entered into... Together with the text encoding Transformer The encoder of the architecture , Pictured 5 Shown . Encode in video frames , So that the model can only use pictures - Text data set for pre training （ Think of pictures as one frame video ）, Then use video for training on downstream tasks . In the pre training phase , The model is modeled only in mask language , And the use of [CLS] Label for visual - Text matching , To learn about cross modal representations .

chart 5 ClipBERT framework ^[6]

3.1.3 VLM

VLM ^[7] Put forward a simple 、 Downstream task independent pre training methods , Unified video 、 The representation of the text , Able to accept video 、 Single mode input of text , Or video - Joint input of text . Pictured 6 Shown , The model introduces mask representation modeling ( Video frame or text token) To create a representation of the interior of the mode ; At the same time, the mask modal modeling task is introduced (MMM), Mask the entire video or the entire text mode at one time , The guidance model utilizes cross modal interactions to recover information .

chart 6 VLM Pre training paradigm ^[7]

3.2 Multi-Stream

3.2.1 CBT

CBT ^[8] Proposed by noisy contrast estimation (noise contrastive estimation, NCE) To learn the video - The representation of the text .CBT Expanded BERT To build a cross modal representation , In the pre training phase , Two single mode Transformer It is used to learn the representation of video and text respectively , And use a cross modal Transform er To build two modules Joint representation of states . Because video features are continuous , For video single mode encoder and cross mode encoder , The model learns its representation through noisy contrast estimation . To be specific , For single mode video coders , The loss function has the following form ：

‍

 

among yes 3D Encoder output , and It's video Transformer Code output of . And for cross modal Transformer Joint representation of , The loss function has the following form ：

among MI(x,y) Delegate will video , Joint hidden layer representation output after joint coding of text modes .

chart 7：CBT The architecture of the model ^[8]

3.2.2 UniVL

Most cross modal models are only used in downstream multimodal understanding tasks , There is not much exploration of cross modal generation tasks .UniVL ^[9] A multimodal framework supporting task generation and task understanding is established . Its architecture is roughly shown in the figure 8 Shown ,UniVL The hidden vector encoded by the single-mode encoder , Input to a cross modal encoder - Decoder architecture . adopt NCE Establish the similarity between different modes , Make the same video - The text is right , The results of different modal encoders have good correlation in representation ; At the same time, the cross modal interaction is established through the cross modal mask language modeling and mask frame modeling . meanwhile , Using decoder to reconstruct text , Introduce cross modal generation capability to the model . Encoder - The introduction of decoder architecture enables the model to be naturally applied to downstream description generation tasks .

chart 8 UniVL Framework 、 Application of pre training tasks and downstream tasks ^[9]

3.2.2 Frozen

Fronzen ^[2] A pre training framework designed for retrieval task is proposed . Think of pictures as video “ snapshot ”, The image and text data set is pre trained as a part of the video data set , The scale of pre training is increased by using graphic data set , Using video at the same time - Text data sets learn to focus on the unique timing information in video content . For efficient cross modal Retrieval , The author uses a single-mode encoder to encode the information of two modes , The information of the two modes is projected into a common representation space , Calculate the similarity . Its architecture is roughly shown in the figure 9 Shown . Video is input in frames , The picture is equivalent to a video that contains only one frame , meanwhile , The model also introduces space-time transformer ^[10] To model spatiotemporal information in video . In the pre training phase ,NCE It is also used to measure the loss of training , The model uses video data and text data alternately for pre training .

chart 9 Frozen The model architecture of [2]

4. Fine grained information in pre training

Compared to pictures , The information contained in video is usually rich and redundant , for example , Between successive frames , action 、 Scenes and other contents may be highly similar . The method based on dense sampling will greatly increase the amount of computation , The sparse sampling method will lose too much fine-grained information . How to extract the fine-grained information needed to understand video from video content , And make use of , Enhance the understanding of video content and the learning of cross modal representation , It has also become one of the issues explored by researchers .

4.1 Based on spatiotemporal information

Compared with the picture , Video content contains more abundant temporal and spatial information , Modeling spatiotemporal information in video , It plays a very important role in understanding video content .

4.1.1 HERO

Less work has been done to explicitly explore the model's understanding of video timing information .HERO ^[11] In single flow cross modal Transformer On the framework of , A timing sequence is introduced Transformer To model the timing information in the video , At the same time, the mask frame is used to predict (MFM) And frame order prediction (FOM) To enhance the model's understanding of timing information in video . To be specific ,MFM The task requires the model to recover the content of the current frame through text information and video context （ Estimate by regression or noisy comparison NCE In the form of ）;FOM Then the frame information after cross modal coding is scrambled proportionally , Required timing Transformer The architecture tries to take advantage of video context and sequential subtitles (subtitle) The information restores the order of the scrambled frames , Pictured 10 Shown .

chart 10 HERO The model architecture of ^[11]

4.1.2 Merlot

Contains dense video descriptions (Dense caption) Video content usually includes rich cross modal temporal alignment information ,Merlot ^[12] Through video frames - Description matches (Contrastive frame-transcript matching) To model the timing information inside the video .Merlot The input of is a sparse sampled video frame , Description corresponding to the video frame , The model maximizes the similarity between the video frame and the corresponding description , Minimize the similarity of video description with other frames in the video to establish timing alignment information between video and text ; And HERO Different ,Merlot Reorder using time series (Temporal Reordering) Mission , Shuffle video frames proportionally , And judge the relative order between frames to guide the model to pay attention to the timing information inside the video .

chart 11 Merlot The architecture of the model ^[12]

4.1.3 DECEMBERT

HowTo100M The aligned text in the dataset usually comes from automatic speech recognition (ASR), Contains more noise , Humans may also describe scenes that have occurred or have not yet spoken , As a result, the timing of text and video clips are not completely aligned , Or semantic inconsistency . Pictured 12 Shown ,DECEMBERT ^[13] From video Clip In the middle, the dense caption To alleviate the problem of noise and semantic inconsistency ; At the same time, the text description of video clip context is introduced , To mitigate possible timing inconsistencies .

chart 12 DECEMBERT The architecture of ^[13]

4.1.4 Violet

Many people think of video as a collection of still pictures , Ignoring the continuity of information in time series , Visual mask tasks designed based on this feature may fail .Violet ^[14] utilize Discrete VAE ^[22] The of the video patch The feature is discretized into a series of visual token, Video coding and text coding joint input cross modal Transformer after , Mask vision token The modeling task requires the model to recover the corresponding discrete vision from the visual mask output token, And mask language modeling tasks are unified . meanwhile , The model introduces block based (Block wise) Discrete visual mask task , Based on time 、 Space masks multiple consecutive positions at the same time , Prevent it from simply recovering the masked information from a spatiotemporal continuous location . Besides , General masking methods mask important with the same probability / Unimportant position ,Violet Introduced Attended Masking Methods , Take advantage of the attention weight across modes , Try to mask areas that the model considers more important , To increase the difficulty of the mask task .

chart 13 Violet Model architecture ^[14]

4.2 Based on entities and actions

be based on patch Or the encoding method of video level features implicitly encodes all kinds of information in the video , Coarse grained video - Text alignment may be difficult to mine the fine-grained information contained in video . The following work also attempts to introduce more fine-grained supervision information such as entities and actions , Enhance cross modal representation learning and modeling .

4.2.1 ActBERT

chart 14 ActBERT The model architecture of ^[15]

Discretize video , Enter with text Transformer The way of joint modeling across modes may lose a lot of fine-grained information ,ActBERT ^[15] By training a video motion coder on the pre training data set , Get the motion code of the video , At the same time, the object detection tool is introduced to obtain the object information coding in the video content . By creating text - action - Entanglement coding architecture between video regions , To establish the interaction between video action, area information and text information , Furthermore, important timing information is retained .

4.2.2 OA-Trans

Multi-Stream Cross modal representation learning usually builds video - Overall alignment between texts , No exploration of finer grained alignment ,OA-Trans ^[16] The entity region information and the corresponding label obtained from the key frame target detection are introduced . Pictured 15 Shown , The model masks the positions in the key frame that do not contain entity regions , After space-time coding, the representation corresponding to the remaining position is obtained , At the same time, the category information of the entity area is input into the text encoder for coding . Simply align the resulting fine-grained representation , Will not be of much help to downstream tasks . Without modifying the model architecture , Researchers try to establish a similarity relationship between a single modal fine-grained representation and another modal global representation , Guiding the overall representation of a single mode can contain more fine-grained information , When applied to downstream tasks , The fine-grained representation used for pre training can be removed , The overall representation of single mode has the ability to retain fine-grained information to a certain extent .

chart 15 OA-Trans The model architecture of ^[16]

4.2.3 ALPRO

The introduction of fine-grained information usually requires the help of external feature extraction tools , This not only affects efficiency , Due to the limitation of the number of categories of feature extraction tools , The effect is not satisfactory . Pictured 16 Shown ,ALPRO ^[17] be based on CLIP ^[18] Thought , First, based on video - Text pairs are trained in a video - Text matching schema ( and ALRP The single-mode encoder has the same architecture ), Describe... Through prompts *[CLS] A video of a [object]*, According to the similarity between video and description , Be able to recognize the entities contained in the video . In the process of training , Model randomly cuts a video , Using the matching architecture to obtain the entity information of the video as the monitoring signal , Prompt entity matching is introduced (Prompt Entity Matching) Mission , It is required that the cross modal encoder can identify the entity information corresponding to the cutting position , So as to establish the video understanding of fine-grained entity information and scene .

chart 16 ALPRO The model architecture of ^[17]

4.2.4 BridgeFormer

Compared with using the information of object detection ,BridgeFormer ^[19] Use multiple choice tasks (Multiple Choice Questions) To enhance the model's understanding of the new entity information and action information in the video . To be specific , Researchers construct by hiding verbs or noun phrases from the original text “ problem ”, The problem representation obtained by the text encoder is regarded as a cross modal Transformer Query for (Query), Encode the video content as a key (Key) And key values (Value), The cross modal representation problem is formalized as a given problem , Query from video , The process of getting answers , When applied to downstream tasks , The single-mode coder has learned how to build a fine-grained representation where cross modal modules can be removed .

chart 17 BridgeFormer The model architecture of ^[19]

4.3 Implicit fine-grained features

Compared with considering explicit fine-grained features , Such as timing 、 Entity 、 Actions and other information , There is also part of the work that in addition to video 、 The overall representation of the text , You can also consider frames 、patch Different levels of granularity represent , To implicitly encode fine-grained features .

4.3.1 HiT

stay Transformer Architecture , Different layers will focus on presentation information of different granularity , For example, lower level attention remembering tends to encode more basic surface representations , And the higher attention mechanism , Will capture more complex semantic information , Based on this idea ,HiT ^[20] A layered cross modal contrast matching mechanism is proposed , To establish different granularity of cross modal representation alignment , To be specific , Models are separately from video 、 Text Query The first and last layers of the encoder obtain the representation of low-level feature level and high-level semantic level . And they are respectively matched with the high-level coding results of another mode , Pictured 18 Shown .

chart 18 HiT The model architecture of ^[20]

4.3.2 HunYuan_tvr

Most retrieval models focus on establishing the representation relationship between the whole video and the whole corresponding text ,HunYuan_tvr ^[21] The relationship between fine-grained representation is explored from multiple levels , By modeling frames - words , Video clip - The phrase , video - Representation matching among three different granularity cross modal representations of a sentence , A hierarchical cross modal interaction method is proposed to learn fine-grained cross modal connections , Roughly as shown in the picture 19 Shown .

chart 19：HunYuan_tvr The general model architecture of ^[21]

5. summary

This article briefly combs the current video - The model architecture of text pre training and related data sets , meanwhile , Aiming at the redundancy of video information , The work of introducing fine-grained information is introduced .

After combing and analyzing, we can find , Current video - The data set of text pre training is difficult to collect and label , Number and size of available datasets and graphs - There is still less text pre training , At the same time, there is a lack of more fine-grained annotations .

To reduce the dependence on computing resources , At the same time, make better use of pictures - Text pre training data , video - The text pre training model gradually transits from dense sampling to sparse sampling , In order to compensate for the information loss caused by sparse sampling and the lack of supervision information of coarse-grained pre training data , A lot of work has begun to explore how to extract 、 Or get useful fine-grained information in an unsupervised way , Further enhance fine-grained video - Text means learning . some time , Build larger scale 、 More granular video - Text pre training data ; Consider more reasonable and useful fine-grained information to help the training process ; Design 、 Take advantage of the more powerful single mode 、 Cross modal model architecture and self supervised learning tasks are both worthy of further exploration .

6. reference

[1] Miech A, Zhukov D, Alayrac J B, et al. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 2630-2640.

[2] Bain M, Nagrani A, Varol G, et al. Frozen in time: A joint video and image encoder for end-to-end retrieval[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 1728-1738.

[3] Sharma P, Ding N, Goodman S, et al. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018: 2556-2565.

[4] Sun C, Myers A, Vondrick C, et al. Videobert: A joint model for video and language representation learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 7464-7473.

[5] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805, 2018.

[6] Lei J, Li L, Zhou L, et al. Less is more: Clipbert for video-and-language learning via sparse sampling[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 7331-7341.

[7] Xu H, Ghosh G, Huang P Y, et al. VLM: Task-agnostic video-language model pre-training for video understanding[J]. arXiv preprint arXiv:2105.09996, 2021.

[8] Sun C, Baradel F, Murphy K, et al. Learning video representations using contrastive bidirectional transformer[J]. arXiv preprint arXiv:1906.05743, 2019.

[9] Luo H, Ji L, Shi B, et al. Univl: A unified video and language pre-training model for multimodal understanding and generation[J]. arXiv preprint arXiv:2002.06353, 2020.

[10] Bertasius G, Wang H, Torresani L. Is space-time attention all you need for video understanding?[C]//ICML. 2021, 2(3): 4.

[11] Li L, Chen Y C, Cheng Y, et al. Hero: Hierarchical encoder for video+ language omni-representation pre-training[J]. arXiv preprint arXiv:2005.00200, 2020.

[12] Zellers R, Lu X, Hessel J, et al. Merlot: Multimodal neural script knowledge models[J]. Advances in Neural Information Processing Systems, 2021, 34: 23634-23651.

[13] Tang Z, Lei J, Bansal M. Decembert: Learning from noisy instructional videos via dense captions and entropy minimization[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021: 2415-2426.

[14] Fu T J, Li L, Gan Z, et al. VIOLET: End-to-end video-language transformers with masked visual-token modeling[J]. arXiv preprint arXiv:2111.12681, 2021.

[15] Zhu L, Yang Y. Actbert: Learning global-local video-text representations[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 8746-8755.

[16] Wang J, Ge Y, Cai G, et al. Object-aware Video-language Pre-training for Retrieval[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 3313-3322.

[17] Li D, Li J, Li H, et al. Align and Prompt: Video-and-Language Pre-training with Entity Prompts[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 4953-4963.

[18] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International Conference on Machine Learning. PMLR, 2021: 8748-8763.

[19] Ge Y, Ge Y, Liu X, et al. Bridging Video-Text Retrieval With Multiple Choice Questions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 16167-16176.

[20] Liu S, Fan H, Qian S, et al. Hit: Hierarchical transformer with momentum contrast for video-text retrieval[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 11915-11925.

[21] Min S, Kong W, Tu R C, et al. HunYuan_tvr for Text-Video Retrivial[J]. arXiv preprint arXiv:2204.03382, 2022.

[22] Van Den Oord A, Vinyals O. Neural discrete representation learning[J]. Advances in neural information processing systems, 2017, 30.