BN folding and its quantification

This paper introduces the process of quantification BN Fold ;

The following pictures are quoted from Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

BN Fold

First introduced BN Fold ;BN Folding comes from BN The implementation difference between the training phase and the inference phase of the module ;

As shown in the figure below ： With conv+BN For example , chart 1 For the training stage BN, chart 2 For the inferential stage BN

chart 1： In the training phase BN

chart 2： Inference stage BN,BN Folded knife CONV in

It can be proved that such folding is reasonable , Pay attention to the picture 2 Medium w It refers to the convolution kernel ;

---

In quantification BN Fold

For quantification , Especially quantitative perception （ Pseudo quantization ）, We need to keep the pseudo quantization in the training stage and the above figure 2 The quantitative implementation of the inference phase of is consistent , Therefore, for the training stage CONV+BN Fold in two steps , Here's the picture 3：

chart 3： In the training phase BN Folding transformation , Ideas come from pictures 2 Implementation shown
​​​

chart 3 Our design idea comes from figure 2, The basic idea is to seek Of EMA（ In the picture moment And the steps before and after ） After taking it off, you should match the picture 2 Agreement ;

According to the figure 3 Shown BN Fold , You can do post training quantification or pseudo quantification of the folding in the training stage , Here's the picture 4

chart 4：BN After folding the training stage （ false ） Quantification process

---

Code implementation

The following code block is from github distiller, There is no complete context implementation , Interested readers can read step by step

chart 5：BN Fold quantization implementation

---

summary

• The above is a brief introduction to BN Folding and its quantification , The thought is quite ingenious , Interested readers can further read in depth according to the literature mentioned in the article , thank you
• Please correct any misunderstandings