当前位置:网站首页>Paper reading [MM21 pre training for video understanding challenge:video captioning with pre training techniqu]
Paper reading [MM21 pre training for video understanding challenge:video captioning with pre training techniqu]
2022-07-07 05:34:00 【hei_ hei_ hei_】
MM21 Pre-training for Video Understanding Challenge:Video Captioning with Pretraining Technique
summary
- publish :ACMM 2021
- idea: Use X-Linear Attention, reference XLAN Right Multi-modality Feature To merge , Propose a multi-path XLAN The model can fuse multiple single-mode features , Get a better fused feature . In addition, in the video understanding pre training model competition, through data expansion technology and integration multi-path XLAN(early fuse) And fine tuning pretrained OPT(late fuse) Get first
Detailed design
1. Single-Modality Pretrained Feature Fusion
Multi-Modality Feature Extraction
Almost all modal features in the video are considered , Include :
(1)appearance feature( 30 f r a m e s ∗ 2048 d i m s 30 frames * 2048 dims 30frames∗2048dims):FixResNeXt-101 network pretrained on the ImageNet-1k dataset
(2)motion feature( 30 f r a m e s ∗ 2048 d i m s 30 frames * 2048 dims 30frames∗2048dims):irCSN-152 network pretrained on the Kinetics-400 dataset
(3)region feature( 50 f r a m e s ∗ 2048 d i m s 50 frames * 2048 dims 50frames∗2048dims):vinvl model pretrained on Visual Genome dataset
(4)audio feature( 30 f r a m e s ∗ 2048 d i m s 30 frames * 2048 dims 30frames∗2048dims):CNN14 network pretrained on the AudioSet datasetMulti-Modality Feature Fusion
The feeling is OPT+XLAN, Almost nothing has changed
F x F_x Fx Represents the input feature , E x E_x Ex It mainly embeds various modal features into the same semantic hidden space , E n c o d e r x Encoder_x Encoderx yes XLAN encoder
there A G G i n AGG_in AGGin and A G G c t x AGG_ctx AGGctx Indicates the aggregation method , There are several options :average pooling、concatenation、additional attention
2. Multi-Modality Pretrained Model Finetuning
Yes pretrained Omni-Perception Pre-Trainer model (OPT) Fine tuning .
- OPT
Use three... Respectively encoder To text 、 picture 、 The sound is encoded and the features are converted to the same latent space; And then use transformer Fuse the three features (inter- and intra interactions), Then access text decoder and visual decoder Generate text and pictures respectively . At the same time token-level、modality-level and sample-level To make the model have the ability of cross modal understanding and generation . The author uses MSR-VTT Fine tune the dataset .
experiment
- Ablation Studies
S P SP SP Direct transfer multi-modality features concate Then proceed reduce dimension To 1024 Then input encoder-decoder Of XLAN/Transformer modal in - Comparison to State-of-the-art
+ R L +RL +RL Indicates that it is used in fine-tuning reinforcement learning
边栏推荐
- How can project managers counter attack with NPDP certificates? Look here
- Flink SQL 实现读写redis,并动态生成Hset key
- 漏电继电器JOLX-GS62零序孔径Φ100
- Two methods of thread synchronization
- Mapbox Chinese map address
- 数字化创新驱动指南
- Mysql database learning (8) -- MySQL content supplement
- Addressable pre Download
- 淘寶商品詳情頁API接口、淘寶商品列錶API接口,淘寶商品銷量API接口,淘寶APP詳情API接口,淘寶詳情API接口
- Dbsync adds support for mongodb and ES
猜你喜欢
随机推荐
JSP setting header information export to excel
纪念下,我从CSDN搬家到博客园啦!
Dj-zbs2 leakage relay
Leetcode 1189 maximum number of "balloons" [map] the leetcode road of heroding
batch size设置技巧
论文阅读【MM21 Pre-training for Video Understanding Challenge:Video Captioning with Pretraining Techniqu】
The navigation bar changes colors according to the route
Scheduledexecutorservice timer
JVM (19) -- bytecode and class loading (4) -- talk about class loader again
Is the human body sensor easy to use? How to use it? Which do you buy between aqara green rice and Xiaomi
Make web content editable
说一说MVCC多版本并发控制器?
When deleting a file, the prompt "the length of the source file name is greater than the length supported by the system" cannot be deleted. Solution
消息队列:消息积压如何处理?
How does redis implement multiple zones?
Creation and use of thread pool
If you want to choose some departments to give priority to OKR, how should you choose pilot departments?
Leetcode (417) -- Pacific Atlantic current problem
Leetcode (46) - Full Permutation
JHOK-ZBL1漏电继电器