当前位置:网站首页>Depth residual network
Depth residual network
2022-07-06 06:54:00 【Programming bear】
AlexNet、 VGG、 GoogLeNet The emergence of neural network model has brought the development of neural network into the stage of dozens of layers , The deeper the layers of the network , The more likely you are to get better generalization ability . But when the model deepens , The Internet is becoming more and more difficult to train , This is mainly due to Gradient dispersion and Gradient explosion Caused by . In deeper layers of neural networks , When the gradient information is transmitted layer by layer from the end layer of the network to the first layer of the network , In the process of transmission The gradient is close to 0 Or a very large gradient
How to solve the phenomenon of gradient dispersion and gradient explosion of deep neural network ? Since the shallow neural network is not prone to gradient phenomenon , that We can try to add a mechanism to the deep neural network to fall back to the shallow neural network . When the deep neural network can easily retreat to the shallow neural network , The deep neural network can obtain the same model performance as the shallow neural network
One 、 Residual network model
By adding a direct connection between input and output Skip Connection It can make the neural network have the ability of fallback
With VGG13 Take deep neural network as an example , Suppose that VGG13 The gradient dispersion phenomenon appears in the model , and 10 The gradient dispersion phenomenon is not observed in the layer network model , Then consider adding... To the last two convolution layers SkipConnection, In this way , The network model can Automatically select whether to pass These two convolution layers complete the feature transformation , still Just skip These two convolution layers are selected Skip Connection, Or combine two convolution layers and Skip Connection Output
be based on Skip Connection Of Deep residual network (Residual Neural Network, abbreviation ResNet) Algorithm , And put forward 18 layer 、 34 layer 、 50 layer 、 101 layer 、 152 Layer of ResNet-18、 ResNet-34、 ResNet-50、 ResNet-101 and ResNet-152 Wait for the model
ResNet By adding Skip Connection Realize the layer fallback mechanism , Input 𝒙 Through two convolution layers , Get the output after feature transformation ℱ(𝒙), With the input 𝒙 Add the corresponding elements , Get the final output ℋ(𝒙):
ℋ(𝒙) = 𝒙 + ℱ(𝒙),ℋ(𝒙) It's called the residual module (Residual Block, abbreviation ResBlock). Because of being Skip Connection Surrounding convolutional neural networks need to learn mapping ℱ(𝒙) = ℋ(𝒙) - 𝒙, So it is called Residual network
To be able to Satisfy input 𝒙 And the output of the convolution layer ℱ(𝒙) Be able to add , Need to enter 𝒙 Of shape And ℱ(𝒙) Of shape Exactly the same . When there is a shape When not in agreement , Usually through Skip Connection Add an additional convolution operation link on the input 𝒙 Change to and ℱ(𝒙) same shape, Pictured identity(𝒙) Function , among identity(𝒙) With 1×1 Most of the convolution operations , It is mainly used to adjust the number of input channels
The depth residual network passes through the stack Residual module , Reached a deeper number of network layers , Thus, the training stability is obtained 、 Deep network model with superior performance
Two 、ResBlock Realization
The deep residual network does not add new network layer types , Just by adding a line between input and output Skip Connection, Not for ResNet The underlying implementation of . stay TensorFlow The residual module can be realized by calling the ordinary convolution layer .
First create a new class , Create the convolution layer needed in the residual block in the initialization phase 、 Activate the function layer
# Residual module class
class BasicBlock(layers.Layer):
def __init__(self, filter_num, stride=1):
super(BasicBlock, self).__init__()
# f(x) It contains two ordinary convolution layers
self.conv1 = layers.Conv2D(filter_num, (3, 3), strides=stride, padding='same')
self.bn1 = layers.BatchNormalization()
self.relu = layers.Activation('relu')
self.conv2 = layers.Conv2D(filter_num, (3, 3), strides=stride, padding='same')
self.bn2 = layers.BatchNormalization()
# f(x) And x Different shapes , Cannot add
if stride != 1: # It's not equal , Insert identity layer
self.downsample = Sequential()
self.downsample.add(layers.Conv2D(filter_num, (1, 1), strides=stride))
else: # Direct additive
self.downsample = lambda x: x
# Forward propagation function
def call(self, inputs, training=None):
out = self.conv1(inputs)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
# Input through identity() transformation
identity = self.downsample(inputs)
# f(x) + x
output = layers.add([out, identity])
# Then activate the function , It's OK to put it in front
output = tf.nn.relu(output)
return output
First new ℱ(𝑥) Convolution layer , When ℱ(𝒙) The shape and shape of 𝒙 Different time , Cannot add directly , We need new identity(𝒙) Convolution layer , To complete 𝒙 Shape conversion . In forward propagation , Only need to ℱ(𝒙) And identity(𝒙) Add up , And add ReLU Activate the function
RseNet By stacking multiple ResBlock It can form a complex deep neural network , Such as ResNet18,ResNet34......
3、 ... and 、DenseNet
DenseNet take Feature map information of all previous layers adopt Skip Connection Aggregate with the current layer output , And ResNet The corresponding positions of are added in different ways , DenseNet Used in the channel shaft 𝑐 Join dimensions , Aggregate feature information
Input 𝑿0 adopt H1 The convolution layer obtains the output 𝑿1, 𝑿1 And 𝑿0 Splice on the channel shaft , Get the characteristic tensor after aggregation , Send in H2 Convolution layer , Get the output 𝑿2, Same method , 𝑿2 Characteristic information of all previous layers 𝑿1 And 𝑿0 Aggregate , Into the next layer . So circular , Until the output of the last layer 𝑿4 And the characteristic information of all previous layers : {𝑿𝑖}𝑖=0, 1, 2, 3 Aggregate to get the final output of the module , Such a be based on Skip Connection Densely connected modules are called Dense Block
DenseNet By stacking multiple Dense Block It can form a complex deep neural network
边栏推荐
- 【软件测试进阶第1步】自动化测试基础知识
- BUU的MISC(不定时更新)
- 18. Multi level page table and fast table
- Fast target recognition based on pytorch and fast RCNN
- 基于购买行为数据对超市顾客进行市场细分(RFM模型)
- 万丈高楼平地起,每个API皆根基
- UDP攻击是什么意思?UDP攻击防范措施
- UWA Pipeline 2.2.1 版本更新说明
- [English] Grammar remodeling: the core framework of English Learning -- English rabbit learning notes (1)
- SQL Server Manager studio (SSMS) installation tutorial
猜你喜欢
mysql的基础命令
Prefix and array series
The internationalization of domestic games is inseparable from professional translation companies
[English] Verb Classification of grammatical reconstruction -- English rabbit learning notes (2)
Lesson 7 tensorflow realizes convolutional neural network
基于PyTorch和Fast RCNN快速实现目标识别
雲上有AI,讓地球科學研究更省力
因高额网络费用,Arbitrum 奥德赛活动暂停,Nitro 发行迫在眉睫
【刷题】怎么样才能正确的迎接面试?
Office doc add in - Online CS
随机推荐
LeetCode每日一题(1870. Minimum Speed to Arrive on Time)
ML之shap:基于adult人口普查收入二分类预测数据集(预测年收入是否超过50k)利用Shap值对XGBoost模型实现可解释性案例之详细攻略
PCL realizes frame selection and clipping point cloud
接口自动化测试实践指导(上):接口自动化需要做哪些准备工作
What is the difference between int (1) and int (10)? Senior developers can't tell!
【Hot100】739. Daily temperature
简单描述 MySQL 中,索引,主键,唯一索引,联合索引 的区别,对数据库的性能有什么影响(从读写两方面)
kubernetes集群搭建Zabbix监控平台
Market segmentation of supermarket customers based on purchase behavior data (RFM model)
Practical guidance for interface automation testing (Part I): what preparations should be made for interface automation
Introduction and underlying analysis of regular expressions
MySQL high frequency interview 20 questions, necessary (important)
How effective is the Chinese-English translation of international economic and trade contracts
将ue4程序嵌入qt界面显示
《从0到1:CTFer成长之路》书籍配套题目(周更)
(practice C language every day) reverse linked list II
Day 246/300 ssh连接提示“REMOTE HOST IDENTIFICATION HAS CHANGED! ”
ROS learning_ Basics
[ 英語 ] 語法重塑 之 動詞分類 —— 英語兔學習筆記(2)
What are the commonly used English words and sentences about COVID-19?