Classic model alexnet

Introduction

To identify more images , We need a model with strong learning ability .CNN Is a good choice . As its depth and width increase , Its performance has also been greatly improved . meanwhile , Compared with feedforward neural network ,CNN With fewer connections and parameters .

With the improvement of computing power ,CNN There is a place for use , That's why we do CNN Why .

This article is based on CNN Designed a new network , contain 5 Convolutions and 3 All connection layers . meanwhile , To avoid overfitting , Using some special techniques .

The Dataset

Cut directly $256\times 256$ Size image , As input .

Milepost ： Directly in raw image Training on .

The Architecture

ReLU Nonlinearity

tanh and sigmoid There is the problem of gradient saturation ,ReLU There is no such problem , So the latter can help the model converge faster .

ReLU It's not necessarily good , It is simple and easy to use .

Local Response Normalization（LRN）

ReLU There is no need to standardize the input to avoid saturation , But still do one Normalization Words , The effect is still improved .

This is because ReLU There is no range , So we should normalize the results .

Create a competitive mechanism for the activity of local neurons , Make the value with larger response become larger , Suppress neurons with small feedback .

（ This part is meaningless ）

Overlapping Pooling

To the traditional Pooling Made a change . Generally, the step size is consistent with the pool size , Here let the step size be smaller than the size , Make the output of the pooling layer overlap , Enhance the richness of features .

Overall Architecture

The figure contains two AlexNet, In the third convolution, the data will be coupled with each other .

Every GPU Training half of the channel .

As the depth of the network increases , Compress the spatial information slowly , At this time, the semantic information increases slowly .

The full connection layer part will also couple the data , Finally, it is spliced as 4096 The vector of the dimension , And then compress it to 1000 dimension , Representative for 1000 Confidence level of two categories .

Just to be able to train , So it is divided into two parts for training , It's not really necessary .

Reduce Overfitting

Data Augmentation

1. from 256x256 Buckle a piece in the middle 224x224 Size image （ He said there was 2048 Seed buckle method , But in fact, they are almost the same ）;
2. hold RGB Make some changes on the channel , It was used PCA Methods .

Dropout

Randomly change some output of the hidden layer to 0, The probability is 50%.

In this article, it is considered that this is a different model obtained from each training and then fusion . But it's not .

It is pointed out that ,Dropout It approximates a linear model L2 The regularization .

In this paper Dropout On top of the two full connection layers .

current CNN Lost the full connection layer in , therefore Dropout Not many are used , But actually , Its presence RNN、Attention Such a fully connected structure is still very common 、 Very easy to use. .

Details of learning

1. SGD;
2. weight decay（ Manually determine when the learning rate decreases epoch, Now it is generally to increase slowly and then decrease slowly ）;
3. momentum（ Avoid that the optimized surface is not so smooth and falls into a pit ）;
4. The standard deviation is 0.01、 The mean for 0 The Gaussian distribution of is used to initialize the weight ; The offset is initialized to 1;

Results

At the end of the article, the effects of the two experiments are shown . The first image shows some pictures and their classification results . The second picture is to AlexNet Finally, take out the eigenvector , Group similar images together . The second experiment found that the result of classification was very good , This illustrates the AlexNet The acquired features are well learned .

summary

AlexNet The contribution of ：

1. Put forward AlexNet;
2. ReLU、LRN、Pooling;
3. Data augmentation （2 Kind of ）;
4. Training details （4 individual ）