Asgnet paper and code interpretation 2

Address of thesis ：Adaptive Prototype Learning and Allocation for Few-Shot Segmentation
Paper code ：ASGNet

Abstract

Prototype learning is widely used in small sample segmentation . Usually , By averaging the global object information , From supporting features （support feature） Get a single prototype in . However , Using a prototype to represent all the information may lead to ambiguity . In this paper , We propose two new modules ： Super pixel guided clustering (SGC) And guided prototype allocation （GPA）, Used for extraction and allocation of various prototypes . say concretely ,SGC It is a method without parameters and training , It extracts more representative prototypes by aggregating similar feature vectors , and GPA Be able to select matching prototypes to provide more accurate guidance . By way of SGC and GPA Bind together , We propose an adaptive super-pixel guidance network (ASGNet), This is a lightweight model , Able to adapt to changes in object size and shape . Besides , Our network can be easily extended to k-shot Division , There are significant improvements and no additional computational costs . special , We use COCO Data set evaluation shows ,ASGNet stay 5-shot The accuracy of segmentation is higher than that of the most advanced methods 5%.

Existing problems and solutions Introduction

current Few-Shot Segmentation networks usually extract features from query images and support images , Then different feature matching methods and target mask transmission methods from support image to query image are proposed . Feature matching and mask passing usually use prototype feature learning technology . Prototype learning technology will support the mask target object of image （masked object features） Compressed into one or several prototype eigenvectors . then , Search the pixel position of similar features in the query image to segment the target .

• One of the main advantages of prototype learning is that prototype features are more robust to noise than pixel features . However , Prototype features inevitably lose spatial information , This is very important when the appearance of objects that support images and query images are quite different . Besides , Most prototype learning networks only generate a single prototype through mask average pooling, thus losing the ability to distinguish information .

In this work , We propose a new prototype learning technology , To address some of the existing major shortcomings . especially , We want to adaptively change the number of prototypes and their spatial range according to the image content , Make the prototype have the ability of content adaptation and spatial awareness . This adaptive multi prototype strategy is very important to deal with the huge changes of object size and shape in different images . Intuitively , When an object occupies a large part of the image , It carries more information , So more prototypes are needed to represent all the necessary information . contrary , If the object is small , The proportion of background is relatively large , Then one or more prototypes are enough . Besides , We want the support area of each prototype ( The scope of space ) It can adapt to the object information appearing in the supporting image . say concretely , Our goal is to divide the supporting features into several representative regions according to feature similarity . We also hope to be able to adaptively select more important prototypes to find more similar features in query images . Different object parts may appear in different image regions and different query images , Therefore, we hope to dynamically allocate different prototypes in the query image for feature matching . for example , Some parts of the object may be occluded in the query image , We want to dynamically select the prototype corresponding to the visible part of the query image .

We use adaptive super pixels to guide the network (ASGNet) To achieve this adaptation 、 Multi archetypal learning and distribution ,ASGNet Use super pixels to adapt to the number of prototypes and support areas . Specially , We propose the composition ASGNet Two modules of the core : Super pixel guided clustering (SGC) And guided prototype allocation (GPA).

• SGC The module carries out feature-based super-pixel fast extraction for supporting images , Got $superimageplainqualityheartdobyprimarytype,sign$. The shape and number of super pixels are adaptive to the image content , Therefore, the generated prototype also becomes adaptive .
• GPA The module uses a mechanism similar to attention to $branchwithBigmanyCountphaseTurn\; offOftheaprimarytype,sign$.

in summary ,SGC The module provides adaptive prototype learning in terms of the number of prototypes and their spatial expansion ,GPA The module provides adaptive allocation of learned prototypes when dealing with query features . These two modules make ASGNet Highly flexible and adaptable to variable object shapes and sizes , Allow it to better generalize invisible object classes .

Proposed Method

In this part , We first introduce two prototype generation and matching modules , That is, the super-pixel guided clustering module （SGC） And guide prototype allocation module （GPA）. then , We discuss the adaptive ability of these two modules . then , We introduced the whole network architecture , It is called adaptive super pixel guidance network （ASGNet）, It will SGC and GPA Modules are integrated in one model . The overall structure is shown in the figure 2 Shown . Last , We explained ASGNet Medium k-shot Set up .

Superpixel-guided Clustering（ Super pixel guided clustering ）

SGC The core idea of super pixel sampling network （SSN）[13] and MaskSLIC[12] Inspired by the .SSN Is the first end-to-end trainable depth network for super pixel segmentation .SSN The key contribution of is to SLIC[1] The nearest neighbor operation in is transformed into differentiable operation . Conventional SLIC The super-pixel algorithm uses k Mean iterative clustering , In two steps ： Pixel super pixel Association and super pixel centroid update . Based on color similarity and proximity , Assign pixels to different superpixel centroids . To be specific , The input image I∈ Rn×5 Usually located with n A five-dimensional space of pixels （labxy）, among lab Express CIELAB Pixel vector in color space ,xy Indicates the pixel position . After iterative clustering , The algorithm outputs the correlation graph , Each of these pixels n Assigned to m One of the super pixels .

This simple method inspired us with a profound idea , That is, the feature map is aggregated into multiple super pixel centroids by clustering , Here the super pixel centroid can be used as a prototype . therefore , We do not calculate the super pixel centroid in the image space , Instead, it is estimated by clustering similar feature vectors , Classify in feature space . Algorithm 1 Describe the whole SGC The process ：