[intensive reading of papers] grounded language image pre training (glip)

One . background

https://arxiv.org/abs/2112.03857
https://github.com/microsoft/GLIP

The task of this paper is phrase grounding, Belong to visual grounding A kind of .phrase grounding My task is to input sentences and pictures , Frame all the objects mentioned in the sentence .visual grounding Other tasks and details can be referred to
https://zhuanlan.zhihu.com/p/388504127

GLIP You can do both target detection and grounding,

• object detection ：
    In the field of amplification target detection SOTA,zero-shot It is better to , You can also do zero-shot Target detection task .
    Compared with conventional target detection tasks, it has rich semantics .
• grounding：
    And Convention grounding Compared with tasks, we can do target detection tasks .

Two 、 contribution

contribution

• Target detection and phrase grounding Tasks are unified for pre training
• Expand visual semantics
• Strong transfer learning ability

performance

• 27M Training on related data . It has strong zero sample and small sample migration performance in target recognition tasks
• Zero-shot：coco val On 49.8AP,LVIS val On 26.9AP
• After fine tuning ：COCO val On 60.8AP
• The downstream 13 Target detection tasks ,1 A sample of GLIP It can be done with Dynamic Head Compete with

3、 ... and 、 Method

3.1 Method 1： Detection and grounding Unified tasks

1. background:
For the detection data set ：
Enter the tag name during training (person、hairdryer）、 box 、 picture .
Input pictures during the test , Predict box and tag names .
The training process is as follows ：

2. background as grounding：
groudning The input to the model is the phrase 、 Boxes and pictures of nouns in phrases .
take object Model to grounding The way to ： adopt prompt To convert tag names into phrases .

Such as coco Yes 80 A label , take 80 Tags are connected by commas , Add “Detect：”, To form short sentences .

The formula 2 Become a formula 3 In the process of ,T The size of will change , from Nc become NM
structure token： In the flow chart above ,M(sub-word tokens) Always better than phrase format c many , There are four reasons 1） Some phrases take up more than toeken Location , such as traffic light.2） Some phrases are separated into sub words, such as toothbrush Divided into tooth#, #brush.3） Some are added token, Like a comma ,Deteckt etc. ,4） The end will add [NoObj] Of token. During training ,phrase If it is a positive example , Multiple subwords They are all positive examples . When testing, there are multiple token The average of pro As a phrase probability.

3. detection and grounding linkage ： By the above method , It can be used grounding Model to pre train detection tasks , So you can migrate GLIP The model makes zero-shot Detection of

3.2 Method 2: deep fusion, The combination of vision and language

fusion Some formulas are as follows ：

​​​​​​​​​​​​​​O⁰ It's vision backbone Of feature, P⁰ It's the text backbone Of feature
X-MHA（cross-modality multi-head attention module）
L yes DyHead in DyHeadModules Number ,BERT Layer Add for .

attention Some are common in multimodal , such as co-attention、guided attention etc. . Refer to multimodal attention Other optimization .
DeepFusion advantage ：
Improved phrase grounding effect
Make visual features language-aware

3.3 Method 3: Pre training with rich semantic data

grounding The semantics of data sets are very rich , Target detection does not exceed 2000 Categories , however grounding Data sets, such as Flickr30K It includes 4.4w Different phrases , The magnitude is different .
How to amplify grounding data ：

1. stay gold data（det+grounding） Train teachers GLIP
2. Use this teacher model to predict 24M web image-text data , adopt NLP Analyze noun phrases , There is 5840 Different noun phrases
3. The student model is gold data And fake tags grounding Training on data
   Amplification effect ：
   The effect of student model is better than that of teacher model , For example, for some words ,vaccine The teacher model may not predict , But it can be predicted a small vial,subwords Right , whole phrase Will be right . When giving students unsupervised data of the model , Can be a small vial of vaccine The whole label is given to the student model as a learning label .
   

Four 、 experimental result

FourODs（2.66M data ） yes 4 A collection of detection data sets , Include objects365、OpenImages、VG Data sets （ except coco）、ImageNetBox.
GoldG+ The dataset includes 1.3M Data sets , Include Flickr30K、VG caption、GQA.
GoldG The dataset is GoldG+ In addition to the coco Data sets

4.1 The migration effect is on the detection data set

zero-shot stay coco On ：

1. Graphic data sets do not bring improvement
2. C and B Greater than the improvement
3. Objects365 It includes coco Of 80 individual
   
   stay LVIS On the effect ：
   LVIS： Large scale fine-grained vocabulary level markup data set ,1000+ Category , Pineapple dices in pizza are also marked
   Gold grounding It works （model C vs model B）
   

4.2 stay grounding On dataset

Flick 30k： Graphic matching grounding Data sets ,goldG This data set is included in

4.3 Ablation Experiment - Detect data set impact

O365: 0.66M
GoldG: 0.8M
FourODs: 2.66M
But it's not O365+GoldG The effect is better

4.4 other

If the positioning is not good , You can add prompt words to help better locate , The figure below adds flat and round