Basics of ResNet: Principles of Residual Blocks

在深度学习中,In order to enhance the learning ability of the model,The network layer will get deeper and deeper,但是随着深度的增加,It also brings up some comparison problems,主要包括：

• Model complexity increases,Network training is difficult;
• 梯度消失/梯度爆炸
• 网络退化,That is to say, the learning ability of the model has reached saturation,Increasing the number of network layers does not improve accuracy anymore.

为了解决网络退化问题,He Kaiming proposed a deep residual network,It can be said that it is a very large creative work in deep learning.

残差网络

The idea of ​​residual network is to map the network learned fromX到YSwitch to learning fromX到Y-X的差,Then add the learned residual information to the original output.even in extreme cases,This residual is0,Then the network is oneX到Y的恒等映射.其示意图如下：

In the structure of the above figure, the main line is actually no different from the normal network structure,The difference is the connecting line on the right,作者称之为Shortcut Connection,This means that some network layers are skipped and directly connected to the output of a subsequent layer.

优势

in the residual network,Because the residual block preserves the information of the original input,So the network has the following advantages：

• 随着深度的增加,Higher precision can be obtained,Because the residual error of its learning is more accurate;
• Network optimization is relatively simple;
• 比较通用;

残差块的实现

Follow the structure shown in the figure above,在PytorchIt is also very simple to implement a residual block in ,It is nothing more than adding one to the traditional networkshortcut connection,For example, a most basic residual block code is as follows：

class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super(ResidualBlock, self).__init__()
        self.conv1 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
        self.bn1 = nn.BatchNorm2d(channels)
        self.prelu = nn.PReLU()
        self.conv2 = nn.Conv2d(channels, channels, kernel_size=3, padding=1)
        self.bn2 = nn.BatchNorm2d(channels)

    def forward(self, x):
        residual = self.conv1(x)
        residual = self.bn1(residual)
        residual = self.prelu(residual)
        residual = self.conv2(residual)
        residual = self.bn2(residual)
        
        out = self.prelu(x + residual)
        return out

One is achieved through the above codeThe most basic residual block（It is only implemented according to the picture,It's not the same as the original）.需要注意的地方有：

• The residual block is because in forwardThe end of the function needs to be the inputxand the learned residuals（也就是 ）相加,So the dimensions of the two tensors should be exactly the same;
• 在最后将 After the addition, enter the activation function;
• Each convolutional layer is followed by a batch normalization layer.

在真正用的时候,The code above needs to be further complicated,For example, whether to downsample the data, etc,But understand the basics above,就可以自己进行相应的修改,to apply to your own network.

参考

【1】HE K, ZHANG X, REN S, et al. Deep Residual Learning for Image Recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2016:770-778. 10.1109/CVPR.2016.90.
【2】https://towardsdev.com/implement-resnet-with-pytorch-a9fb40a77448