当前位置：网站首页>Knowledge distilling learning notes

Knowledge distilling learning notes

2022-06-29 21:01:00 【Study hard, depeng】

【 Classic Jane read 】 Distillation of knowledge (Knowledge Distillation) A classic
paper
Huangzhenhua , Yangshunzhi , Lin Wei , Nijuan , Sunshengli , Chen Yunwen , Tang Yong . A review of knowledge distillation [J]. Journal of Computer Science ,2022,45(03):624-653.
【 intensive reading AI The paper 】 Distillation of knowledge

One 、MMRazor

Model compression toolbox

Insert picture description here

Two 、MMDeploy

Model deployment toolkit

Insert picture description here

3、 ... and 、 Distillation of knowledge

Insert picture description here

Soft Target Contains more information

Predicted by teacher network Soft Target As a label Train students to network

Insert picture description here

Insert picture description here

Knowledge Distilling and Label Smoothing

Insert picture description here

Development trend of knowledge distillation

Insert picture description here

Question1: Will knowledge distillation improve the accuracy of the model

Answer1: Knowledge distillation is a new method to obtain efficient and small-scale networks , Its main idea is to integrate the learning ability into the complex teacher model “ knowledge ” Move to a simple student model . meanwhile , It learns from each other through neural networks 、 Optimization strategies such as self-learning and tagless 、 Cross modal and other data resources also have a significant effect on the performance enhancement of the model .
Insert picture description here