Paper reading [MM21 pre training for video understanding challenge:video captioning with pre training techniqu]

MM21 Pre-training for Video Understanding Challenge:Video Captioning with Pretraining Technique

summary

• publish ：ACMM 2021
• idea： Use X-Linear Attention, reference XLAN Right Multi-modality Feature To merge , Propose a multi-path XLAN The model can fuse multiple single-mode features , Get a better fused feature . In addition, in the video understanding pre training model competition, through data expansion technology and integration multi-path XLAN（early fuse） And fine tuning pretrained OPT（late fuse） Get first

Detailed design

1. Single-Modality Pretrained Feature Fusion

• Multi-Modality Feature Extraction
  Almost all modal features in the video are considered , Include ：
  （1）appearance feature（$30frames\ast 2048dims$）：FixResNeXt-101 network pretrained on the ImageNet-1k dataset
  （2）motion feature（$30frames\ast 2048dims$）：irCSN-152 network pretrained on the Kinetics-400 dataset
  （3）region feature（$50frames\ast 2048dims$）：vinvl model pretrained on Visual Genome dataset
  （4）audio feature（$30frames\ast 2048dims$）：CNN14 network pretrained on the AudioSet dataset

• Multi-Modality Feature Fusion
  The feeling is OPT+XLAN, Almost nothing has changed
  
  
  $F_x$ Represents the input feature ,$E_x$ It mainly embeds various modal features into the same semantic hidden space ,$Encode{r}_x$ yes XLAN encoder
  
  there $AG{G}_{i}n$ and $AG{G}_{c}tx$ Indicates the aggregation method , There are several options ：average pooling、concatenation、additional attention

2. Multi-Modality Pretrained Model Finetuning

Yes pretrained Omni-Perception Pre-Trainer model (OPT) Fine tuning .

• OPT
  
  Use three... Respectively encoder To text 、 picture 、 The sound is encoded and the features are converted to the same latent space; And then use transformer Fuse the three features （inter- and intra interactions）, Then access text decoder and visual decoder Generate text and pictures respectively . At the same time token-level、modality-level and sample-level To make the model have the ability of cross modal understanding and generation . The author uses MSR-VTT Fine tune the dataset .

experiment

• Ablation Studies
  
  $SP$ Direct transfer multi-modality features concate Then proceed reduce dimension To 1024 Then input encoder-decoder Of XLAN/Transformer modal in
• Comparison to State-of-the-art
  
  $+RL$ Indicates that it is used in fine-tuning reinforcement learning