ECCV 2022 | open source for generative knowledge distillation of classification, detection and segmentation

The article links ： https://arxiv.org/abs/2205.01529

Code link ： https://github.com/yzd-v/MGD

Knowledge distillation can be mainly divided into logit Distillation and feature Distillation . among feature Distillation has better expansibility , It has been applied in many visual tasks . However, due to the different model structures of different tasks , many feature Distillation method is designed for a specific task .

Previous knowledge distillation methods focused on making students imitate the characteristics of stronger teachers , So that students' characteristics have stronger representation ability . We believe that improving students' representation ability does not necessarily need to be achieved by directly imitating teachers . From this point of view , We changed the imitation task into a generation task ： Let students rely on their weak characteristics to generate teachers' strong characteristics . In distillation , We made a random analysis of students' characteristics mask, Force students to use only some of their own characteristics to generate all the characteristics of teachers , To improve students' representation ability .

In order to prove MGD Not by imitating teachers to improve students , We have visualized the characteristic map of students and teachers . You can see , The attention of students and teachers before distillation is very different . In the use of FGD Distillation （ Imitate teachers ） after , Students' attention becomes very close to teachers , Performance has also been greatly improved . But when using MGD After distillation , There are great differences between students and teachers , Students' response to the background is greatly reduced , The response to the target is enhanced , Students' final performance is better than FGD Distillation .