Depth residual network

AlexNet、 VGG、 GoogLeNet The emergence of neural network model has brought the development of neural network into the stage of dozens of layers , The deeper the layers of the network , The more likely you are to get better generalization ability . But when the model deepens , The Internet is becoming more and more difficult to train , This is mainly due to Gradient dispersion and Gradient explosion Caused by . In deeper layers of neural networks , When the gradient information is transmitted layer by layer from the end layer of the network to the first layer of the network , In the process of transmission The gradient is close to 0 Or a very large gradient

How to solve the phenomenon of gradient dispersion and gradient explosion of deep neural network ？ Since the shallow neural network is not prone to gradient phenomenon , that We can try to add a mechanism to the deep neural network to fall back to the shallow neural network . When the deep neural network can easily retreat to the shallow neural network , The deep neural network can obtain the same model performance as the shallow neural network
 

One 、 Residual network model

By adding a direct connection between input and output Skip Connection It can make the neural network have the ability of fallback

With VGG13 Take deep neural network as an example , Suppose that VGG13 The gradient dispersion phenomenon appears in the model , and 10 The gradient dispersion phenomenon is not observed in the layer network model , Then consider adding... To the last two convolution layers SkipConnection, In this way , The network model can Automatically select whether to pass These two convolution layers complete the feature transformation , still Just skip These two convolution layers are selected Skip Connection, Or combine two convolution layers and Skip Connection Output

be based on Skip Connection Of Deep residual network (Residual Neural Network, abbreviation ResNet) Algorithm , And put forward 18 layer 、 34 layer 、 50 layer 、 101 layer 、 152 Layer of ResNet-18、 ResNet-34、 ResNet-50、 ResNet-101 and ResNet-152 Wait for the model

ResNet By adding Skip Connection Realize the layer fallback mechanism , Input 𝒙 Through two convolution layers , Get the output after feature transformation ℱ(𝒙), With the input 𝒙 Add the corresponding elements , Get the final output ℋ(𝒙)：
ℋ(𝒙) = 𝒙 + ℱ(𝒙),ℋ(𝒙) It's called the residual module (Residual Block, abbreviation ResBlock). Because of being Skip Connection Surrounding convolutional neural networks need to learn mapping ℱ(𝒙) = ℋ(𝒙) - 𝒙, So it is called Residual network

To be able to Satisfy input 𝒙 And the output of the convolution layer ℱ(𝒙) Be able to add , Need to enter 𝒙 Of shape And ℱ(𝒙) Of shape Exactly the same . When there is a shape When not in agreement , Usually through Skip Connection Add an additional convolution operation link on the input 𝒙 Change to and ℱ(𝒙) same shape, Pictured identity(𝒙) Function , among identity(𝒙) With 1×1 Most of the convolution operations , It is mainly used to adjust the number of input channels

The depth residual network passes through the stack Residual module , Reached a deeper number of network layers , Thus, the training stability is obtained 、 Deep network model with superior performance
 

  Two 、ResBlock Realization

The deep residual network does not add new network layer types , Just by adding a line between input and output Skip Connection, Not for ResNet The underlying implementation of . stay TensorFlow The residual module can be realized by calling the ordinary convolution layer .

First create a new class , Create the convolution layer needed in the residual block in the initialization phase 、 Activate the function layer

#  Residual module class 
class BasicBlock(layers.Layer):
    def __init__(self, filter_num, stride=1):
        super(BasicBlock, self).__init__()
        # f(x) It contains two ordinary convolution layers 
        self.conv1 = layers.Conv2D(filter_num, (3, 3), strides=stride, padding='same')
        self.bn1 = layers.BatchNormalization()
        self.relu = layers.Activation('relu')

        self.conv2 = layers.Conv2D(filter_num, (3, 3), strides=stride, padding='same')
        self.bn2 = layers.BatchNormalization()

        # f(x)  And  x  Different shapes , Cannot add 
        if stride != 1:  #  It's not equal , Insert identity layer 
            self.downsample = Sequential()
            self.downsample.add(layers.Conv2D(filter_num, (1, 1), strides=stride))
        else:  #  Direct additive 
            self.downsample = lambda x: x

    #  Forward propagation function 
    def call(self, inputs, training=None):
        out = self.conv1(inputs)
        out = self.bn1(out)
        out = self.relu(out)
        out = self.conv2(out)
        out = self.bn2(out)
        #  Input through identity() transformation 
        identity = self.downsample(inputs)
        # f(x) + x
        output = layers.add([out, identity])
        #  Then activate the function , It's OK to put it in front 
        output = tf.nn.relu(output)
        return output

First new ℱ(𝑥) Convolution layer , When ℱ(𝒙) The shape and shape of 𝒙 Different time , Cannot add directly , We need new identity(𝒙) Convolution layer , To complete 𝒙 Shape conversion . In forward propagation , Only need to ℱ(𝒙) And identity(𝒙) Add up , And add ReLU Activate the function
 

RseNet By stacking multiple ResBlock It can form a complex deep neural network , Such as ResNet18,ResNet34......

3、 ... and 、DenseNet

DenseNet take Feature map information of all previous layers adopt Skip Connection Aggregate with the current layer output , And ResNet The corresponding positions of are added in different ways , DenseNet Used in the channel shaft 𝑐 Join dimensions , Aggregate feature information

Input 𝑿0 adopt H1 The convolution layer obtains the output 𝑿1, 𝑿1 And 𝑿0 Splice on the channel shaft , Get the characteristic tensor after aggregation , Send in H2 Convolution layer , Get the output 𝑿2, Same method , 𝑿2 Characteristic information of all previous layers 𝑿1 And 𝑿0 Aggregate , Into the next layer . So circular , Until the output of the last layer 𝑿4 And the characteristic information of all previous layers ： {𝑿𝑖}𝑖=0, 1, 2, 3 Aggregate to get the final output of the module ,  Such a be based on Skip Connection Densely connected modules are called Dense Block

DenseNet By stacking multiple Dense Block It can form a complex deep neural network