常见神经网络参数量与计算量

一、卷积层

注意 使用Batch Normalization时不需要bias 每个输出通道一个数值作为bias

参数量
P = Cout*Cin*K*K+bias
bias = Cout

计算量
步幅为1
乘 加 偏置
FlOPs = [(Cin*k*k)+(Cin*k*k-1)+1]*Cout*W*H
记乘加为浮点运算
FLOPs = Cin*k*k*Cout*W*H
若步幅不为1 可以用输出图片宽高作为WH带入近似

二、全连接

参数量
P = In*Out+bias
bias = Out

计算量
乘 加 偏置
FLOPs = [In+(In-1)+1]*Out
记乘加为浮点运算
FLOPs = In*Out

三、循环神经网络

注意双向要翻倍

参数量
x为输入维度
h为隐层维度
PRNN = (x+h)*h+bias
bias = h
PGRU = 3*PRNN
PLSTM = 4*PRNN

四、注意力

参数量
P = 4*(dim*Innerdim)
Innerdim = headdim*heads

计算量
[B,N,dim]->[B,N,Innerdim]
[B,N,Innerdim]->[B,heads,N,Innerdim/heads]
FLOPs = N*dim*Innerdim*3
Q*K'
FLOPs = heads*N*(Innerdim/heads)*N = N*N*Innerdim
Q*K'*V
FLOPs = heads*N*N*(Innerdim/heads) = N*N*Innerdim
FLOPs = N*dim*Innerdim
总共
FLOPs = 4*N*dim*Innerdim + 2*N*N*Innerdim = 2*N*Innerdim*(2*dim+N)