当前位置:网站首页>BN折叠及其量化
BN折叠及其量化
2022-07-06 08:51:00 【cyz0202】
本文介绍量化过程中的BN折叠;
以下图片引用自 Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
BN折叠
首先介绍BN折叠;BN折叠源自 BN模块在训练阶段和推断阶段的实现差异;
如下图所示:以conv+BN为例,图1为训练阶段的BN,图2为推断阶段的BN
可以证明这样的折叠是合理的,注意图2中的w指的是卷积核;
量化中的BN折叠
对量化来说,尤其是量化感知(伪量化),我们需要保持训练阶段的伪量化和上述图2的推断阶段的量化实现是一致的,因此对训练阶段的CONV+BN二阶段步骤进行折叠,如下图3:
图3的设计思想源自图2,基本思想是训练阶段求的EMA(即图中moment及其前后步骤)拿掉后要和图2一致;
根据图3所示BN折叠,就可以对训练阶段该折叠做后训练量化或者伪量化,如下图4
代码实现
以下代码块来自github distiller,没有完整的上下文实现,感兴趣的读者可以移步阅读
总结
- 以上简要介绍了BN折叠及其量化,思想比较巧妙,感兴趣的读者可以根据文中提到的文献进一步深入阅读,谢谢
- 理解有误的地方欢迎指正
边栏推荐
- LeetCode:387. The first unique character in the string
- Computer graduation design PHP Zhiduo online learning platform
- UML圖記憶技巧
- LeetCode:498. Diagonal traversal
- Image, CV2 read the conversion and size resize change of numpy array of pictures
- The ECU of 21 Audi q5l 45tfsi brushes is upgraded to master special adjustment, and the horsepower is safely and stably increased to 305 horsepower
- Generator parameters incoming parameters
- LeetCode:673. 最长递增子序列的个数
- Leetcode刷题题解2.1.1
- LeetCode:836. 矩形重叠
猜你喜欢
深度剖析C语言数据在内存中的存储
Swagger setting field required is mandatory
UnsupportedOperationException异常
Problems encountered in connecting the database of the project and their solutions
TCP/IP协议
vb.net 随窗口改变,缩放控件大小以及保持相对位置
Sublime text using ctrl+b to run another program without closing other runs
注意力机制的一种卷积替代方式
[OC]-<UI入门>--常用控件-UIButton
Warning in install. packages : package ‘RGtk2’ is not available for this version of R
随机推荐
如何正确截取字符串(例:应用报错信息截取入库操作)
LeetCode:41. 缺失的第一个正数
[OC]-<UI入门>--常用控件-UIButton
数学建模2004B题(输电问题)
Target detection - pytorch uses mobilenet series (V1, V2, V3) to build yolov4 target detection platform
[OC-Foundation框架]-<字符串And日期与时间>
@JsonBackReference和@JsonManagedReference(解决对象中存在双向引用导致的无限递归)
The ECU of 21 Audi q5l 45tfsi brushes is upgraded to master special adjustment, and the horsepower is safely and stably increased to 305 horsepower
LeetCode:836. 矩形重叠
UnsupportedOperationException异常
Sublime text in CONDA environment plt Show cannot pop up the problem of displaying pictures
Leetcode刷题题解2.1.1
@Jsonbackreference and @jsonmanagedreference (solve infinite recursion caused by bidirectional references in objects)
[sword finger offer] serialized binary tree
Double pointeur en langage C - - modèle classique
Excellent software testers have these abilities
目标检测——Pytorch 利用mobilenet系列(v1,v2,v3)搭建yolov4目标检测平台
多元聚类分析
Deep analysis of C language data storage in memory
Tcp/ip protocol