ECCV 2022 | ssp: a new idea of small sample tasks with self-supporting matching

author | Fan Qi

Company | Hong Kong University of science and technology

Research direction | Small sample learning / Model generalization

This is where we are ECCV 2022 New job on , for the first time （to my best knowledge） Self supporting matching is proposed in small sample tasks （self-support matching） Thought , And use it in small sample segmentation （Few-shot semantic segmentation, FSS） Tasks , With very novel ideas and very simple methods, good results have been achieved , No additional parameters , The model is very small , Training and testing are fast . At the same time, we hope to apply our self-supporting matching idea to other related tasks . The code is open source .

Paper title ：

Self-Support Few-Shot Semantic Segmentation

Thesis link ：

https://arxiv.org/abs/2207.11549

Code link ：

https://github.com/fanq15/SSP

▲ chart 1

Annotation

• ： Support pictures （support image）

• ： Support image features （support feature）

• ： Support feature prototypes （support prototype）

• ： Search for pictures （query image）

• ： Query image features （query feature）

• ： Query feature prototype （query prototype）, Self supporting prototype （self-support prototype）

FSS Task

For a new category of interested targets , Small sample segmentation algorithm through the given support image For query pictures Target new category in . among , For each new category of goals , There is no appearance or mark in the training set , And the number of samples corresponding to the test is only a few pictures （ Generally lower than 10 Zhang ）.

The classic small sample segmentation algorithm is to support images For query pictures Extract features from , And make use of Masked Average Pooling Operation and support GT mask In support of image features Generate support feature prototypes on . Finally, calculate the supporting feature prototype And query image features Cosine similarity between each spatial location feature （cosine similarity）, And will the cosine similarity map As the predicted segmentation result . The specific process is shown in the figure 0 Shown on the left .

▲ chart 0

FSS Problem

There is a very important problem in the small sample segmentation task , Namely support and query Between the appearance gap. stay support and query Similar objects in may have very large appearance differences . For example, the same category is cats , But different cat breeds 、 The posture and the lighting conditions for taking photos may be completely different , Pictured 0 The picture on the right shows . This problem stems from the characteristics of small sample tasks , That is, data scarcity and data diversity .

Data scarcity refers to support There's very little data , Usually, each new category has less than 10 Samples , Data diversity refers to query The number of is endless . therefore support The objects in cannot cover all query Similar objects in . Pictured 0 Shown on the left ,support It's Garfield's cat head ,query It's a whole black cat , The traditional matching algorithm can only segment the black cat's head , Because the two have semantic similarity , But Garfield cat's head is quite different from black cat's body , Therefore, a satisfactory segmentation result cannot be obtained .

Other Solutions

The previous method also noticed this key problem and tried to solve it . Some methods try to learn better support feature prototypes ： Use unlabeled images to get better support for feature prototypes [1,2], Or propose a complex prototype generation module for supporting features , Such as using attention mechanism [3,4] And adaptive learning [5,6,7] etc. . There are also some ways to propose better training methods [8,9] To improve the matching effect . But these methods cannot fundamentally solve support and query Between appearance gap problem . Because they are still limited to the use of very few support To divide infinite query.

Our Idea

We propose a new self-supporting small sample segmentation algorithm . The idea is to use query feature prototypes To match query image features . Among them, query feature prototype Using high confidence query prediction mask Extract the corresponding query image features Got , and query prediction mask By tradition support-query Generated by matching algorithm . Because of this query Self matching properties of features , We name the method self-supporting （self-support） Algorithm .

Our Motivation

The idea of self-supporting matching comes from Gestalt theory （Gestalt principle）, That is, the integrity of the object ： Relative to different parts of different objects , Different parts of the same object will be more like . To test this idea , We counted Pascal VOC The similarity between different pixels of the image in the dataset , It is characterized by ImageNet Pre trained ResNet-50 Extracted .

As shown in the table 1 Shown , For foreground pixels , The similarity between foreground pixels on the same object is much greater than that between foreground pixels on different objects . We also observed the same phenomenon on the background pixels . But what is verified here is the relationship between pixels , Our goal is to prototype features support prototype And query image features query feature A match between . So we use another experiment to further verify our self-supporting idea .

As shown in the table 2 Shown , We use traditional support feature prototypes （ from support Extracted from the picture support prototype） And self-supporting feature prototypes （ from query Extracted from the picture self-support prototype, Because we know query GT mask）, Respectively and query image features Match . Self supporting feature prototype The matching result can reach 83.0 mIoU, Support feature prototypes The matching result of is high and close 25 A little bit .

But in practice , We can't know query GT mask, So we artificially set different sizes query GT mask Noise is introduced to simulate the predicted query mask. under these circumstances , Self supporting feature prototype The matching result of is still higher than 74 mIoU. These two experiments well support the idea of self-supporting matching .

▲ surface 1

▲  surface 2

Method

The method part is very simple , Pictured 2 Shown , We pass the traditional support prototype based matching Generate initial query mask, And make use of query mask Querying image features Extract self-supporting feature prototype on . Finally, the self-supporting feature prototype And support feature prototypes Integrate and query image features Match . At the same time, we can carry out multiple self-supporting matching on self-supporting feature prototypes To optimize .

▲ chart 2

Another Problem and Solution

It can be noted that , Our algorithms are respectively right foreground and background Of prototype Modeling , however foreground and background It's a totally different nature . about foreground Come on , Different parts of the same object are generally similar , Have global similarity , So you can use masked average pooling Will all foreground Pixels are aggregated into one prototype Vector to represent foreground.

But for background Come on , The background is generally messy , There is no global similarity , Only local similarity , So let's use a prototype Vector to represent background There is a problem . Therefore, based on background The local similarity of , We proposed adaptive self-support background prototype（ASBP）, Based on the current background Pixels and others background Similarity between pixels , Weighted fusion background features . The model framework is shown in the figure 3 Shown .

▲  chart 3

Experiments

We are Pascal VOC,MS COCO and FSS-1000 Experiments were conducted on three data sets . As shown in the table 3 And table 4 Shown , In the use of ResNet-101 backbone and 5-shot support Set up , Our method achieves the best performance , At the same time, our method is also very competitive under other settings . We found that HSNet Of evaluation protocol It's easier to MS COCO Get better results on the dataset , So we also use HSNet Of evaluation protocol To evaluate our algorithm .

▲  surface 3

▲  surface 4

Analysis

1. chart 4 Visualize the workflow and effect of self-supporting matching , You can see support and query The object in has great appearance gap, But our self-supporting matching can be well utilized self-support prototype Get accurate segmentation results . Here we take 0.8 As a starting point query mask The threshold of , To show for generating self-support prototype Of query feature.

2. surface 5 The ablation experiment proved the effectiveness of our modules .

3. chart 5 Respectively for mask threshold The value of ASBP The working principle of is visualized .

4. surface 6 Verified the difference prototype The effect of .

5. surface 7 The difference between self support algorithm and self attention method is discussed .

▲  chart 4

▲  surface 5

▲  chart 5

▲  surface 6

▲  surface 7

Advantages

Our self-supporting algorithm has many advantages .

First of all One advantage is that self-supporting algorithms will benefit more from better backbone and support Number , Because they can bring more accurate initial query mask Predicted results , It can extract more accurate self-support prototype, And get better segmentation results . This is a Matthew effect , So our method is ResNet-101 backbone and 5-shot support It will be much better than other algorithms . At the same time 8 It also shows the performance improvement of self-supporting algorithm under different conditions .

second One advantage is the higher confidence of the prediction results . The segmentation algorithm will use a threshold to predict the continuity in the result [0,1] Value to 0-1 value , But the quality of the initial prediction results is also important , Especially for difficult samples . If the predicted value is 0.5 near , Predictions at this point are easy to fail . We hope to get more robust and high confidence prediction results . therefore , We use MAE Continuity of prediction results [0,1] Value . Pictured 9 Shown , Our algorithm can get lower MAE, This means that our prediction results have a higher confidence . surface 9 And table 10 Show our algorithm for other forms support Annotations are also robust .

Third One advantage is that self-supporting algorithms are very general , There is no additional parameter quantity , And the extra computation is also very small , The model is very small , Training and testing are fast , And it can be directly applied to other small sample segmentation algorithms . As shown in the table 11 Shown , We use self-supporting algorithms in PANet and PPNet On , Have been improved .

▲  surface 8

▲  surface 9, surface 10, surface 11

Story

The idea of self-supporting matching is that I'm DDL Thought of before , Start doing it immediately , But what I did at that time didn't work , So I put it on hold for a few days . But I think this idea must be feasible , So I went back and rethinked the whole method , I found that I was using query mask selection query feature There was a problem when . I started with 0.5 Select as the threshold query feature Of , But it will introduce more noise . Later, I will try to raise the threshold to 0.7, The method works immediately .

therefore The key point of self support is to choose as many high confidence query feature To produce high quality self-support prototype. The whole project took a total of 6 weeks , It's easy , But the idea is novel , I hope so few-shot Inspired by the field . The idea of self-supporting matching is in few-shot The field is rarely or never explored . Self supporting ideas can be extended to other few-shot Tasks . because support and query Between appearance gap yes few-shot Common and important problems in .

Acknowledgment

Thank you for your guidance , thank Lihe Yang @LeolhYang Good job MLC [1] And provide a strong baseline, Thank you for your support , Thank my little Garfield, Feifei, for playing teaser In the picture support. Finally, I wish everyone can get a job SSP.

reference

[1] Mining latent classes for few-shot segmentation (ICCV 2021)

[2] Part-aware prototype network for few-shot semantic segmentation (ECCV 2020)

[3] Simpropnet: Improved similarity propagation for few-shot image segmentation (IJCAI 2020)

[4] Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning (CVPR 2019)

[5] Adaptive prototype learning and allocation for few-shot segmentation (CVPR 2021)

[6] Self-supervision with superpixels: Training few-shot medical image segmentation without annotation (ECCV 2020)

[7] Amp: Adaptive masked proxies for few-shot segmentation (ICCV 2019)

[8] Crnet: Cross-reference networks for few-shot segmentation (CVPR 2020)

[9] Panet: Few-shot image semantic segmentation with prototype alignment (ICCV 2019)

Read more

# cast draft   through Avenue #

  Let your words be seen by more people  

How to make more high-quality content reach the reader group in a shorter path , How about reducing the cost of finding quality content for readers ？ The answer is ： People you don't know .

There are always people you don't know , Know what you want to know .PaperWeekly Maybe it could be a bridge , Push different backgrounds 、 Scholars and academic inspiration in different directions collide with each other , There are more possibilities . 

PaperWeekly Encourage university laboratories or individuals to , Share all kinds of quality content on our platform , It can be Interpretation of the latest paper , It can also be Analysis of academic hot spots 、 Scientific research experience or Competition experience explanation etc. . We have only one purpose , Let knowledge really flow .

  The basic requirements of the manuscript ：

• The article is really personal Original works , Not published in public channels , For example, articles published or to be published on other platforms , Please clearly mark  

• It is suggested that  markdown  Format writing , The pictures are sent as attachments , The picture should be clear , No copyright issues

• PaperWeekly Respect the right of authorship , And will be adopted for each original first manuscript , Provide Competitive remuneration in the industry , Specifically, according to the amount of reading and the quality of the article, the ladder system is used for settlement

  Contribution channel ：

• Send email ：[email protected] 

• Please note your immediate contact information （ WeChat ）, So that we can contact the author as soon as we choose the manuscript

• You can also directly add Xiaobian wechat （pwbot02） Quick contribution , remarks ： full name - contribute

△ Long press add PaperWeekly Small make up

Now? , stay 「 You know 」 We can also be found

Go to Zhihu home page and search 「PaperWeekly」

Click on 「 Focus on 」 Subscribe to our column

·

·