当前位置:网站首页>Depth residual network
Depth residual network
2022-07-06 06:54:00 【Programming bear】
AlexNet、 VGG、 GoogLeNet The emergence of neural network model has brought the development of neural network into the stage of dozens of layers , The deeper the layers of the network , The more likely you are to get better generalization ability . But when the model deepens , The Internet is becoming more and more difficult to train , This is mainly due to Gradient dispersion and Gradient explosion Caused by . In deeper layers of neural networks , When the gradient information is transmitted layer by layer from the end layer of the network to the first layer of the network , In the process of transmission The gradient is close to 0 Or a very large gradient
How to solve the phenomenon of gradient dispersion and gradient explosion of deep neural network ? Since the shallow neural network is not prone to gradient phenomenon , that We can try to add a mechanism to the deep neural network to fall back to the shallow neural network . When the deep neural network can easily retreat to the shallow neural network , The deep neural network can obtain the same model performance as the shallow neural network
One 、 Residual network model
By adding a direct connection between input and output Skip Connection It can make the neural network have the ability of fallback
With VGG13 Take deep neural network as an example , Suppose that VGG13 The gradient dispersion phenomenon appears in the model , and 10 The gradient dispersion phenomenon is not observed in the layer network model , Then consider adding... To the last two convolution layers SkipConnection, In this way , The network model can Automatically select whether to pass These two convolution layers complete the feature transformation , still Just skip These two convolution layers are selected Skip Connection, Or combine two convolution layers and Skip Connection Output
be based on Skip Connection Of Deep residual network (Residual Neural Network, abbreviation ResNet) Algorithm , And put forward 18 layer 、 34 layer 、 50 layer 、 101 layer 、 152 Layer of ResNet-18、 ResNet-34、 ResNet-50、 ResNet-101 and ResNet-152 Wait for the model
ResNet By adding Skip Connection Realize the layer fallback mechanism , Input 𝒙 Through two convolution layers , Get the output after feature transformation ℱ(𝒙), With the input 𝒙 Add the corresponding elements , Get the final output ℋ(𝒙):
ℋ(𝒙) = 𝒙 + ℱ(𝒙),ℋ(𝒙) It's called the residual module (Residual Block, abbreviation ResBlock). Because of being Skip Connection Surrounding convolutional neural networks need to learn mapping ℱ(𝒙) = ℋ(𝒙) - 𝒙, So it is called Residual network
To be able to Satisfy input 𝒙 And the output of the convolution layer ℱ(𝒙) Be able to add , Need to enter 𝒙 Of shape And ℱ(𝒙) Of shape Exactly the same . When there is a shape When not in agreement , Usually through Skip Connection Add an additional convolution operation link on the input 𝒙 Change to and ℱ(𝒙) same shape, Pictured identity(𝒙) Function , among identity(𝒙) With 1×1 Most of the convolution operations , It is mainly used to adjust the number of input channels
The depth residual network passes through the stack Residual module , Reached a deeper number of network layers , Thus, the training stability is obtained 、 Deep network model with superior performance
Two 、ResBlock Realization
The deep residual network does not add new network layer types , Just by adding a line between input and output Skip Connection, Not for ResNet The underlying implementation of . stay TensorFlow The residual module can be realized by calling the ordinary convolution layer .
First create a new class , Create the convolution layer needed in the residual block in the initialization phase 、 Activate the function layer
# Residual module class
class BasicBlock(layers.Layer):
def __init__(self, filter_num, stride=1):
super(BasicBlock, self).__init__()
# f(x) It contains two ordinary convolution layers
self.conv1 = layers.Conv2D(filter_num, (3, 3), strides=stride, padding='same')
self.bn1 = layers.BatchNormalization()
self.relu = layers.Activation('relu')
self.conv2 = layers.Conv2D(filter_num, (3, 3), strides=stride, padding='same')
self.bn2 = layers.BatchNormalization()
# f(x) And x Different shapes , Cannot add
if stride != 1: # It's not equal , Insert identity layer
self.downsample = Sequential()
self.downsample.add(layers.Conv2D(filter_num, (1, 1), strides=stride))
else: # Direct additive
self.downsample = lambda x: x
# Forward propagation function
def call(self, inputs, training=None):
out = self.conv1(inputs)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
# Input through identity() transformation
identity = self.downsample(inputs)
# f(x) + x
output = layers.add([out, identity])
# Then activate the function , It's OK to put it in front
output = tf.nn.relu(output)
return output
First new ℱ(𝑥) Convolution layer , When ℱ(𝒙) The shape and shape of 𝒙 Different time , Cannot add directly , We need new identity(𝒙) Convolution layer , To complete 𝒙 Shape conversion . In forward propagation , Only need to ℱ(𝒙) And identity(𝒙) Add up , And add ReLU Activate the function
RseNet By stacking multiple ResBlock It can form a complex deep neural network , Such as ResNet18,ResNet34......
3、 ... and 、DenseNet
DenseNet take Feature map information of all previous layers adopt Skip Connection Aggregate with the current layer output , And ResNet The corresponding positions of are added in different ways , DenseNet Used in the channel shaft 𝑐 Join dimensions , Aggregate feature information
Input 𝑿0 adopt H1 The convolution layer obtains the output 𝑿1, 𝑿1 And 𝑿0 Splice on the channel shaft , Get the characteristic tensor after aggregation , Send in H2 Convolution layer , Get the output 𝑿2, Same method , 𝑿2 Characteristic information of all previous layers 𝑿1 And 𝑿0 Aggregate , Into the next layer . So circular , Until the output of the last layer 𝑿4 And the characteristic information of all previous layers : {𝑿𝑖}𝑖=0, 1, 2, 3 Aggregate to get the final output of the module , Such a be based on Skip Connection Densely connected modules are called Dense Block
DenseNet By stacking multiple Dense Block It can form a complex deep neural network
边栏推荐
- SQL Server Manager studio (SSMS) installation tutorial
- Attributeerror successfully resolved: can only use cat accessor with a ‘category‘ dtype
- 《从0到1:CTFer成长之路》书籍配套题目(周更)
- [English] Grammar remodeling: the core framework of English Learning -- English rabbit learning notes (1)
- [ 英語 ] 語法重塑 之 動詞分類 —— 英語兔學習筆記(2)
- 基于购买行为数据对超市顾客进行市场细分(RFM模型)
- Delete external table source data
- Apache DolphinScheduler源码分析(超详细)
- What are the commonly used English words and sentences about COVID-19?
- BUU的MISC(不定时更新)
猜你喜欢
On the first day of clock in, click to open a surprise, and the switch statement is explained in detail
ROS2安装及基础知识介绍
[brush questions] how can we correctly meet the interview?
我的创作纪念日
SQL Server Manager studio (SSMS) installation tutorial
kubernetes集群搭建Zabbix监控平台
Basic commands of MySQL
Introduction to ros2 installation and basic knowledge
接口自动化测试框架:Pytest+Allure+Excel
【服务器数据恢复】IBM服务器raid5两块硬盘离线数据恢复案例
随机推荐
BUU的MISC(不定时更新)
Bitcoinwin (BCW): the lending platform Celsius conceals losses of 35000 eth or insolvency
AI on the cloud makes earth science research easier
kubernetes集群搭建Zabbix监控平台
[brush questions] how can we correctly meet the interview?
Attributeerror successfully resolved: can only use cat accessor with a ‘category‘ dtype
基于PyTorch和Fast RCNN快速实现目标识别
攻防世界 MISC中reverseMe简述
Delete external table source data
Erreur de type résolue avec succès: type de données « catégorie» non sous - jacente
Leetcode daily question (1870. minimum speed to arrive on time)
女生学软件测试难不难 入门门槛低,学起来还是比较简单的
ROS2安装及基础知识介绍
C语言_双创建、前插,尾插,遍历,删除
P5706 [deep foundation 2. Example 8] redistributing fat house water -- February 13, 2022
【每日一题】729. 我的日程安排表 I
pymongo获取一列数据
The ECU of 21 Audi q5l 45tfsi brushes is upgraded to master special adjustment, and the horsepower is safely and stably increased to 305 horsepower
My seven years with NLP
GET 和 POST 请求类型的区别