Pytorch parameter initialization

feasible initializer Yes kaiming_normal、xavier_normal.

• Kaiming Initialization The paper
• xavier The paper

Reference blog weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming

Gradient explosion and gradient disappearance

Back propagation

Chain according to gradient Back propagation , When the deep gradient is going to spread to the shallow , Will be multiplied by the gradient of this layer . We refer to Deep feedforward network and Xavier Initialization principle Give examples . Suppose there is a linear connection layer followed by an activation layer , Here's the picture ：

• Linear connection layer f2 The input is x, Output is z. namely z = f 2