"Target detection" + "visual understanding" realizes the understanding of the input image

Put forward GLIPv2, Based on the VL Understanding model of , It serves localization Mission （ for example , object detection 、 Instance segmentation ） And visual language （VL） Understand the task （ for example ,VQA、 Image caption ）.

Address of thesis ：https://arxiv.org/pdf/2206.05836.pdf

Code address ：https://github.com/microsoft/GLIP

The minimum of the pre training model 2.5G,

01 summary

GLIPv2 Gracefully localization Pre training and visual language pre training (VLP) Combined with three pre training tasks ：phrase grounding As a detection task VL restructure , Area - Word contrast learning as a new area - Word level contrastive learning task and mask language modeling . This unification not only simplifies the previous multi-stage VLP Program , And it realizes the mutual benefit between positioning and understanding tasks . Experimental results show that , Single GLIPv2 Model （ All model weights are shared ） The approach has been realized on various positioning and understanding tasks SoTA Performance of . The model also shows ：

• Strong zero sample and small sample adaptive performance on open vocabulary target detection task ;

• stay VL Understand the excellence of the task grounding Ability

02 background

lately , People generally pay attention to the construction of general vision system , Also known as visual basic model , It can solve various visual tasks at the same time , For example, image classification 、 Object detection , And visual language (VL) understand . Of particular interest is the positioning task （ for example , Target detection and segmentation ） and VL Understand the task （ for example ,VQA And image captions ） Unity between .

localization Pre training is good for VL Mission ,“localization->VLP” The two-stage pre training process is VL Community . A long-standing challenge is localization And understanding , It aims to achieve mutual benefit between these two tasks , Simplify pre training procedures and reduce pre training costs .

However , The two tasks seem very different ： The positioning task is only a visual task , Fine grained output is required （ for example , Bounding box or pixel mask ）, and VL Understanding tasks emphasizes the integration between the two modes , High level semantic output is required . for example , Answer or title ）.

03 New framework

Left: GLIPv2, a pre-trained grounded VL understanding model, unifies various localization and VL understanding tasks. These two kinds of tasks mutually benefit each other, and enables new capabilities such as language-guided detection/segmentation and grounded VQA/captioning. Right: Additional examples from ODinW (detection), LVIS (segmentation), VQA, and COCO Captioning.

A Unified VL Formulation and Architecture

GLIPv2 The core of the unified formula is the classification matching technique , It rephrases any task specific fixed vocabulary classification problem as a task independent open vocabulary visual language matching problem . The best example is in CLIP Rephrase image classification as image - Text matching , This enables the model to be directly extracted from the original image - Learning from text data , And achieve strong zero sample results on the open vocabulary classification task . stay GLIPv2 in , We use visual language matching point product layer to replace each semantic classification linear layer in the traditional unimodal visual model .

GLIPv2 Pre-training

GLIPv2 Use three pre training losses for pre training ： Visual language reconstruction from target detection task phrase grounding Loss Lground、 Regional word contrast loss from the new regional word level contrast learning task Linter, And standard mask BERT Language modeling loss proposed in Lmlm.

Transfer GLIPv2 to Localization and VL Tasks

We have introduced two easy ways to GLIPv2 Methods of transmitting to various downstream tasks . Besides ,GLIPv2 Traditional VL Mission （ for example VQA）, Effectively make every task we think become “ The basis of VL understand ” Mission .

GLIPv2 pre-training losses: the intra-image alignment loss Lintra (right) takes features after VL fusion and compute loss over region-word pairs within each image-text pair; the inter-image contrastive loss (left) Linter takes features before VL fusion and compute loss over all region-word pairs across a batch of image-text pairs. Label propagation is used to determine the off-diagonal blocks of the Linter target matrix.

04

Experiment and visualization