Mxnet implementation of network in Nin network

         Address of thesis ：NIN(Network In Network, Network in network ), It literally means nesting many network modules ( Micro network ), The structure of this network model is to integrate multi-layer perceptron (Multilayer Perceptron,MLP) And convolution layer , Using multi-layer perceptron to instantiate micro network , Is a strongly efficient function approximator , Stack such structures , To achieve the depth of NIN, Followed by a maximum pooling layer , Reduce the original input sample by half .

         Last in Softmax Global average pooling is used on the characteristic graph of the classification layer (Global Average Pooling,GAP) Get the input , It can get all the feature map information very well , Make spatial transformation more robust .
         Secondly, the global average pooling layer replaces the full connection layer , We know that MXNet Multichannel input and multichannel output ,1x1 Convolution of Use 1*1 Convolution kernel can be equivalent to full connection layer , The advantage of this replacement is that the model is not easy to over fit , And there is no fixed requirement for the input image size . The paper also points out that GAP It is easy to explain and understand the consistency between classification and feature map , The full connectivity layer looks like a “ Black box ”, Not easy to understand , Also very dependent on Dropout Regularization .
The following figure contains three MLPCONV Add a layer GAP Layer of NIN, At every MLPCONV Layer , There is a three-layer perceptron , among NIN And the layers of micro networks can be adjusted by themselves , More flexible .

structure NiN block

NiN Blocks are similar to VGG block ( Yes VGGNet The implementation of the (3*3 The repeated use of convolution kernel ))
A convolution layer with its own set of hyperparameters plus two fixed hyperparameters 1*1 The convolution of layer （ Equivalent to full connection layer ）

def nin_block(num_channels,kernel_size,strides,padding):
    blk=nn.Sequential()
    blk.add(nn.Conv2D(num_channels,kernel_size,strides,padding,activation='relu'),
            nn.Conv2D(num_channels,kernel_size=1,activation='relu'),
            nn.Conv2D(num_channels,kernel_size=1,activation='relu'))
    return blk

Such a nin The block is finished , Since it is NIN A network model , Then there will be many such blocks stacked , We can model the network as before , Add such... To the model nin It's just a piece ,

structure NIN Model

import d2lzh as d2l
from mxnet import gluon,init,nd
from mxnet.gluon import nn

def nin_block(num_channels,kernel_size,strides,padding):
    blk=nn.Sequential()
    blk.add(nn.Conv2D(num_channels,kernel_size,strides,padding,activation='relu'),
            nn.Conv2D(num_channels,kernel_size=1,activation='relu'),
            nn.Conv2D(num_channels,kernel_size=1,activation='relu'))
    return blk

net=nn.Sequential()
net.add(nin_block(96,kernel_size=11,strides=4,padding=0),
        nn.MaxPool2D(pool_size=3,strides=2),
        nin_block(256,kernel_size=5,strides=1,padding=2),
        nn.MaxPool2D(pool_size=3,strides=2),
        nin_block(384,kernel_size=3,strides=1,padding=1),
        nn.MaxPool2D(pool_size=3,strides=2),
        nn.Dropout(0.5),
        nin_block(10,kernel_size=3,strides=1,padding=1),
        # Global average pooling can automatically set the window shape to the input height and width 
        nn.GlobalAvgPool2D(),
        # 4-D output to 2-D output , Shape is ( Batch size ,10)
        nn.Flatten())

# Observe the output shape of each layer 
X=nd.random.uniform(shape=(5,1,224,224))
net.initialize()
for n in net:
    X=n(X)
    print(n.name,' Shape of the output ：',X.shape)

'''
sequential1  Shape of the output ： (5, 96, 54, 54)
pool0  Shape of the output ： (5, 96, 26, 26)
sequential2  Shape of the output ： (5, 256, 26, 26)
pool1  Shape of the output ： (5, 256, 12, 12)
sequential3  Shape of the output ： (5, 384, 12, 12)
pool2  Shape of the output ： (5, 384, 5, 5)
dropout0  Shape of the output ： (5, 384, 5, 5)
sequential4  Shape of the output ： (5, 10, 5, 5)
pool3  Shape of the output ： (5, 10, 1, 1)
flatten0  Shape of the output ： (5, 10)
'''

Training models

We still use Fashion-MNIST Data sets to train , You can compare VGG, The code is as follows ：

# Training models （ Use Fashion-MNIST Data sets ）
lr,num_epochs,batch_size,ctx=0.1,5,128,d2l.try_gpu()
net.initialize(force_reinit=True,ctx=ctx,init=init.Xavier())
trainer=gluon.Trainer(net.collect_params(),'sgd',{'learning_rate':lr})
train_iter,test_iter=d2l.load_data_fashion_mnist(batch_size,resize=96)#224 out of memory 
d2l.train_ch5(net,train_iter,test_iter,batch_size,trainer,ctx,num_epochs)
'''
epoch 1, loss 2.0787, train acc 0.254, test acc 0.514, time 40.4 sec
epoch 2, loss 1.2241, train acc 0.555, test acc 0.664, time 37.9 sec
epoch 3, loss 0.9790, train acc 0.644, test acc 0.669, time 38.2 sec
epoch 4, loss 0.9049, train acc 0.679, test acc 0.703, time 38.2 sec
epoch 5, loss 0.8572, train acc 0.708, test acc 0.748, time 38.3 sec
'''