Neural network parameter initialization

1. bn Parameter initialization of layer

bn The parameters that the layer needs to initialize are scale and bias,bias It is usually initialized to 0,scale There are two ways to initialize

• Initialize to 1
• Initialize to 0： It is assumed that bn Layer does not exist , It is generally used in structures with residuals ：resblock, Because there is chortcut, So this initialization method is feasible

# Zero-initialize the last BN in each residual branch,
# so that the residual branch starts with zeros, and each residual block behaves like an identity.
# This improves the model by 0.2~0.3% according to https://arxiv.org/abs/1706.02677
if zero_init_residual:
    for m in self.modules():
        if isinstance(m, Bottleneck):
            nn.init.constant_(m.bn3.weight, 0)
        elif isinstance(m, BasicBlock):
            nn.init.constant_(m.bn2.weight, 0)

2. fc Parameter initialization of layer

• weight： Normal distribution to initialize , The mean for 0, Variance can be adjusted
• bias：0.01 Or for 0, It's usually 0

if isinstance(m, nn.Linear):
    nn.init.normal_(m.weight, 0, 0.01)  # The mean for 0,  The variance of 0.01
    nn.init.zeros_(m.bias)

3. conv Parameter initialization of layer

• bias：0.01 Or for 0, It's usually 0
• weight：

（1）Xavier initialization ：Xavier Parameter initialization 、 Deep learning parameter initialization （ One ）Xavier initialization With code

（2）He initialization ： Deep learning parameter initialization （ Two ）Kaiming initialization With code

stay He During initialization, you can choose mode by :fan_in perhaps fan_out, Some bloggers concluded ：

• If the weight is through linear layer （ Convolution or full connection ） Implicitly determined , You need to set mode=fan_in;
• If by creating random matrix Explicitly create weights , Then set mode=‘fan_out’.

I looked at the major mainstream models, such as mobilenet_v2,resnet They all use mode=‘fan_out’