Pytorch learning notes 5: model creation

In the previous section, I sorted out the knowledge points of the data module , In this section, we mainly focus on how to use pytorch Build a model to expand , Last use pytorch Realization Alexnet The construction of network structure .

Next, we will explore the implementation details of each module based on the above framework .

One 、 Model creation

It's on it LetNet Network diagram of , It's made up of edges and nodes , The node represents the size of the input data , And edges are operations between data . From the above LetNet Can be seen in the network , The network accepts an input , Then an output is obtained through operation , Inside the network structure , It is also divided into several sub network layers for splicing , The splicing between these sub network layers coordination , Finally, we get the output we want .

So through the above analysis , You can get two elements of building the model ：
● Building sub modules （ For example, the convolution layer in the network structure 、 Pooling layer 、 Activation layer 、 Fully connected layer ）;
● Splice submodules （ Splice the sub modules in a certain order , Finally get the network structure you want ）.

Construct with the task of RMB classification LetNet Network structure ：

class LeNet(nn.Module):
    def __init__(self, classes):
        super(LeNet, self).__init__()

        self.conv1 = nn.Conv2d(3, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16*5*5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, classes)

    def forward(self, x):
        # layer1: conv2d -> relu -> max_pool
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2)

        # layer2: conv2d -> relu -> max_pool
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2)

        # layer3: fc1 -> relu
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))

        # layer4: fc2 -> relu
        x = F.relu(self.fc2(x))

        # layer5:fc3
        x = self.fc3(x)

        return x

    def initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.xavier_normal_(m.weight.data)
                if m.bias is not None:
                    m.bias.data.zero_()
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.data.fill_(1)
                m.bias.data.zero_()
            elif isinstance(m, nn.Linear):
                nn.init.normal_(m.weight.data, 0, 0.1)
                m.bias.data.zero_()

From the code above, we can see ,LeNet Class inherited nn.Module, And in __init__ The method realizes the construction of each sub module , So the first element of building a model —— The construction sub module is embodied here .
How to realize the splicing of sub modules ？——LetNet The inside of the class forward Method , Let's debug the code , See how to call forward Method to realize the splicing operation of sub modules .

Main program model training part , stay outputs = net(inputs) Place a breakpoint , Because this is the part where the model starts training , Here is also the process of the model starting to propagate forward ,debug as follows ：

Program entry module.py In the document __call_impl function , This is because of the class LetNet Inherited nn.Module. Here we will call forward Method , Put the sub modules together .

Inherited in the construction model nn.Module class , All network layers are inherited from this class . So let's take a closer look nn.Module class .

Two 、nn.Module class

1、nn.Module An introduction to the

torch.nn: This is a Pytorch The neural network module of , there Module It's one of its submodules , There are also a few with Module Parallel submodules . among nn.Module The customization of is ：

class Module(object):
    def __init__(self):
        torch._C._log_api_usage_once("python.nn_module")

        self.training = True
        self._parameters = OrderedDict()
        self._buffers = OrderedDict()
        self._non_persistent_buffers_set = set()
        self._backward_hooks = OrderedDict()
        self._forward_hooks = OrderedDict()
        self._forward_pre_hooks = OrderedDict()
        self._state_dict_hooks = OrderedDict()
        self._load_state_dict_pre_hooks = OrderedDict()
        self._modules = OrderedDict()

    def register_parameter(self, name: str, param: Optional[Parameter]) -> None:
    def add_module(self, name: str, module: Optional['Module']) -> None:
    def get_parameter(self, target: str) -> "Parameter":
    def apply(self: T, fn: Callable[['Module'], None]) -> T:
    def cuda(self: T, device: Optional[Union[int, device]] = None) -> T:
    def cpu(self: T) -> T:
    def state_dict(self, destination=None, prefix='', keep_vars=False):
    def load_state_dict(self, state_dict: 'OrderedDict[str, Tensor]',
                        strict: bool = True):
    def named_parameters(self, prefix: str = '', 
                         recurse: bool = True) -> Iterator[Tuple[str, Parameter]]:
    def children(self) -> Iterator['Module']:
    def named_children(self) -> Iterator[Tuple[str, 'Module']]:
    def modules(self) -> Iterator['Module']:
    def named_modules(self, memo: Optional[Set['Module']] = None, 
                      prefix: str = '', remove_duplicate: bool = True):
    def train(self: T, mode: bool = True) -> T:
    def eval(self: T) -> T:
    ...
"""  There are other functions  """

stay nn.Module Properties in customization ：
● parameters： Storage management nn.Parameter class ;
● modules： Storage management nn.Modules class ;
● buffers： Storage management buffer properties , Such as BN Layer. running_mean;
● ***_hook： Storage management hook function

In the construction of network structure, we need to inherit nn.Module class , And reconstruct __init__ and forward Two methods , But there are some things that need to be noticed ：
● Generally, the layer with learnable parameters in the network （ Such as convolution 、 Fully connected layer ） Put it in the constructor __init__ in , Of course, you can also put layers without parameters inside ;
● Generally, layers that do not have learnable parameters ( Such as ReLU、dropout、BatchNormanation layer ) Can be placed in the constructor , It can also be left out of the constructor , If you don't put it in the constructor __init__ Inside , It's in forward You can use nn.functional Instead of ;
● forward Methods must be rewritten , It is the function of implementing the model , The core of realizing the connection relationship between various layers .

for example ：
1、 Put all the layers in the constructor __init__ in , And then in forward Connect these layers in order .

import torch.nn as nn

class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()

        self.conv1 = nn.Conv2d(3, 32, 3, 1, 1)
        self.relu1 = nn.ReLU()
        self.max_pooling1 = nn.MaxPool2d(2, 1)

        self.conv2 = nn.Conv2d(32, 32, 3, 1, 1)
        self.relu2 = nn.ReLU()
        self.max_pooling2 = nn.MaxPool2d(2, 1)
        
        self.dense1 = nn.Linear(32*3*3, 128)
        self.dense2 = nn.Linear(128, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = self.relu1(x)
        x = self.max_pooling1(x)
        x = self.conv2(x)
        x = self.relu2(x)
        x = self.max_pooling2(x)
        x = self.dense1(x)
        x = self.dense2(x)
        return x

model = MyNet()
print(model)

Output results ：

2、 Put the layer without training parameters into forward in , So these layers do not appear in model in , But the running relationship is forward Pass through torch.nn.function Realization , as follows ：

import torch.nn.functional as F

class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()

        self.conv1 = nn.Conv2d(3, 32, 3, 1, 1)
        self.conv2 = nn.Conv2d(3, 32, 3, 1, 1)

        self.dense1 = nn.Linear(32 * 3 * 3, 128)
        self.dense2 = nn.Linear(128, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = F.max_pool2d(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x)
        x = self.dense1(x)
        x = self.dense2(x)
        return x

model = MyNet()
print(model)

Output results ：

As can be seen from the above two examples , In the constructor __init__ The layer in this model “ Intrinsic properties ”.

2、nn.Module The implementation of the

2.1 adopt Sequential Packing layer
Pack several layers into one large layer （ block ）, Then directly call the wrapped layer .Sequential Is defined as follows ：

class Sequential(Module): #  Inherit Module
    def __init__(self, *args):  #  Overriding the constructor 
    def _get_item_by_idx(self, iterator, idx):
    def __getitem__(self, idx):
    def __setitem__(self, idx, module):
    def __delitem__(self, idx):
    def __len__(self):
    def __dir__(self):
    def forward(self, input):  #  Rewrite key methods forward

Add ：
Sequential Three ways of implementation are as follows ：
1、 The simplest sequential model

import torch.nn as nn

model = nn.Sequential(
                      nn.Conv2d(3, 24, 3),
                      nn.ReLU(),
                      nn.Conv2d(24, 5, 3),
                      nn.ReLU())
print(model)

Output results ：

Sequential(
  (0): Conv2d(3, 24, kernel_size=(3, 3), stride=(1, 1))
  (1): ReLU()
  (2): Conv2d(24, 5, kernel_size=(3, 3), stride=(1, 1))
  (3): ReLU()
)

As can be seen from the output above , Each layer of the model has no name , The default is 0,1,2,3 Named after .
2、 Add a name to each layer

import torch.nn as nn
from collections import OrderedDict
model = nn.Sequential(OrderedDict([
                  ('conv1', nn.Conv2d(1,20,5)),
                  ('relu1', nn.ReLU()),
                  ('conv2', nn.Conv2d(20,64,5)),
                  ('relu2', nn.ReLU())
                ]))
 
print(model)
print(model[2]) #  Get the layers through the index 
"""  Output results ： Sequential( (conv1): Conv2d(1, 20, kernel_size=(5, 5), stride=(1, 1)) (relu1): ReLU() (conv2): Conv2d(20, 64, kernel_size=(5, 5), stride=(1, 1)) (relu2): ReLU() ) Conv2d(20, 64, kernel_size=(5, 5), stride=(1, 1)) """

As can be seen from the output above , The output of each layer has a name , But you can't get the layer by name , Still get it by index , namely model[2] That's right. , and model[‘conv2’] It's wrong. . This can actually be seen from Sequential The definition of , Only support index visit .

3、Sequential The third implementation of

import torch.nn as nn
from collections import OrderedDict

model = nn.Sequential()
model.add_module("conv1",nn.Conv2d(1,20,5))
model.add_module('relu1', nn.ReLU())
model.add_module('conv2', nn.Conv2d(20,64,5))
model.add_module('relu2', nn.ReLU())
 
print(model)
print(model[2]) #  Get the layers through the index 

”“”
 Output results ：
Sequential(
  (conv1): Conv2d(1, 20, kernel_size=(5, 5), stride=(1, 1))
  (relu1): ReLU()
  (conv2): Conv2d(20, 64, kernel_size=(5, 5), stride=(1, 1))
  (relu2): ReLU()
)
Conv2d(20, 64, kernel_size=(5, 5), stride=(1, 1))
“”“

add_module The method is Sequential Inherited the parent class Module Methods .

because sequential Can be achieved in the above three ways , The packaging of layers can also be realized in three ways
Mode one ：

import torch.nn as nn
from collections import OrderedDict
class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.conv_block = nn.Sequential(
            nn.Conv2d(3, 32, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(2))
        self.dense_block = nn.Sequential(
            nn.Linear(32 * 3 * 3, 128),
            nn.ReLU(),
            nn.Linear(128, 10)
        )
    #  The connection between layers is realized here , In fact, it is the so-called forward communication 
    def forward(self, x):
        conv_out = self.conv_block(x)
        res = conv_out.view(conv_out.size(0), -1)
        out = self.dense_block(res)
        return out
 
model = MyNet()
print(model)

''' The running result is ： MyNet( (conv_block): Sequential( (0): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): ReLU() (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) ) (dense_block): Sequential( (0): Linear(in_features=288, out_features=128, bias=True) (1): ReLU() (2): Linear(in_features=128, out_features=10, bias=True) ) ) '''

Mode two ：

import torch.nn as nn
from collections import OrderedDict
class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.conv_block = nn.Sequential(
            OrderedDict(
                [
                    ("conv1", nn.Conv2d(3, 32, 3, 1, 1)),
                    ("relu1", nn.ReLU()),
                    ("pool", nn.MaxPool2d(2))
                ]
            ))
 
        self.dense_block = nn.Sequential(
            OrderedDict([
                ("dense1", nn.Linear(32 * 3 * 3, 128)),
                ("relu2", nn.ReLU()),
                ("dense2", nn.Linear(128, 10))
            ])
        )
 
    def forward(self, x):
        conv_out = self.conv_block(x)
        res = conv_out.view(conv_out.size(0), -1)
        out = self.dense_block(res)
        return out
 
model = MyNet()
print(model)

''' The running result is ： MyNet( (conv_block): Sequential( (conv1): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (relu1): ReLU() (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) ) (dense_block): Sequential( (dense1): Linear(in_features=288, out_features=128, bias=True) (relu2): ReLU() (dense2): Linear(in_features=128, out_features=10, bias=True) ) ) '''

Mode three ：

import torch.nn as nn
from collections import OrderedDict
class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.conv_block=torch.nn.Sequential()
        self.conv_block.add_module("conv1",torch.nn.Conv2d(3, 32, 3, 1, 1))
        self.conv_block.add_module("relu1",torch.nn.ReLU())
        self.conv_block.add_module("pool1",torch.nn.MaxPool2d(2))
 
        self.dense_block = torch.nn.Sequential()
        self.dense_block.add_module("dense1",torch.nn.Linear(32 * 3 * 3, 128))
        self.dense_block.add_module("relu2",torch.nn.ReLU())
        self.dense_block.add_module("dense2",torch.nn.Linear(128, 10))
 
    def forward(self, x):
        conv_out = self.conv_block(x)
        res = conv_out.view(conv_out.size(0), -1)
        out = self.dense_block(res)
        return out
 
model = MyNet()
print(model)

''' The running result is ： MyNet( (conv_block): Sequential( (conv1): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (relu1): ReLU() (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) ) (dense_block): Sequential( (dense1): Linear(in_features=288, out_features=128, bias=True) (relu2): ReLU() (dense2): Linear(in_features=128, out_features=10, bias=True) ) ) '''

3、nn.Module Use of several common methods in

Sequenrtial Class implements integer indexing , Therefore, it can be used model[index] Get a layer in this way , however Module Class does not implement integer indexing , It is not possible to obtain the layer by integer index , So what should we do ？ It provides several main methods , as follows ：

def children(self):
 
def named_children(self):
 
def modules(self):
 
def named_modules(self, memo=None, prefix=''):
 
'''  Be careful ： These methods return a Iterator iterator , Therefore, it was passed for Loop access , Of course, it can also be next '''

Take the network constructed above as an example to illustrate as follows ：
1、model.children() Method

import torch.nn as nn
from collections import OrderedDict
class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.conv_block=torch.nn.Sequential()
        self.conv_block.add_module("conv1",torch.nn.Conv2d(3, 32, 3, 1, 1))
        self.conv_block.add_module("relu1",torch.nn.ReLU())
        self.conv_block.add_module("pool1",torch.nn.MaxPool2d(2))
 
        self.dense_block = torch.nn.Sequential()
        self.dense_block.add_module("dense1",torch.nn.Linear(32 * 3 * 3, 128))
        self.dense_block.add_module("relu2",torch.nn.ReLU())
        self.dense_block.add_module("dense2",torch.nn.Linear(128, 10))
 
    def forward(self, x):
        conv_out = self.conv_block(x)
        res = conv_out.view(conv_out.size(0), -1)
        out = self.dense_block(res)
        return out
 
model = MyNet()
 
for i in model.children():
    print(i)
    print(type(i)) #  See what type of elements are in each iteration , It's actually  Sequential  type , So you can use subscripts index Index to get each Sequenrial  The concrete layer inside 
 
# The running result is ：
Sequential(
  (conv1): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (relu1): ReLU()
  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
)
<class 'torch.nn.modules.container.Sequential'>
Sequential(
  (dense1): Linear(in_features=288, out_features=128, bias=True)
  (relu2): ReLU()
  (dense2): Linear(in_features=128, out_features=10, bias=True)
)
<class 'torch.nn.modules.container.Sequential'>

2、model.named_children

for i in model.named_children():
    print(i)
    

 # Output results ：
 ('conv_block', Sequential(
  (conv1): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (relu1): ReLU()
  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
))
('dense_block', Sequential(
  (dense1): Linear(in_features=288, out_features=128, bias=True)
  (relu2): ReLU()
  (dense2): Linear(in_features=128, out_features=10, bias=True)
))

summary ：
（1）model.children() and model.named_children() Method returns an iterator iterator;

（2）model.children(): Each element returned from each iteration is actually Sequential type , and Sequential Type can also use subscript index Index to get each Sequenrial The concrete layer inside , such as conv layer 、dense Layer, etc. ;

（3）model.named_children(): Each element returned from each iteration is actually A tuple type , The first element of a tuple is the name , The second element is the corresponding layer or Sequential.

3、model.modules() Method

for i in model.modules():
    print(i)
    print("==================================================")
#  Output results ：
MyNet(
  (conv_block): Sequential(
    (conv1): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu1): ReLU()
    (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (dense_block): Sequential(
    (dense1): Linear(in_features=288, out_features=128, bias=True)
    (relu2): ReLU()
    (dense2): Linear(in_features=128, out_features=10, bias=True)
  )
)
==================================================
Sequential(
  (conv1): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (relu1): ReLU()
  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
)
==================================================
Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
==================================================
ReLU()
==================================================
MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
==================================================
Sequential(
  (dense1): Linear(in_features=288, out_features=128, bias=True)
  (relu2): ReLU()
  (dense2): Linear(in_features=128, out_features=10, bias=True)
)
==================================================
Linear(in_features=288, out_features=128, bias=True)
==================================================
ReLU()
==================================================
Linear(in_features=128, out_features=10, bias=True)
==================================================

4、model.named_modules() Method

for i in model.named_modules():
    print(i)
    print("==================================================")
    
#  Output results ：
('', MyNet(
  (conv_block): Sequential(
    (conv1): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu1): ReLU()
    (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (dense_block): Sequential(
    (dense1): Linear(in_features=288, out_features=128, bias=True)
    (relu2): ReLU()
    (dense2): Linear(in_features=128, out_features=10, bias=True)
  )
))
==================================================
('conv_block', Sequential(
  (conv1): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (relu1): ReLU()
  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
))
==================================================
('conv_block.conv1', Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)))
==================================================
('conv_block.relu1', ReLU())
==================================================
('conv_block.pool1', MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False))
==================================================
('dense_block', Sequential(
  (dense1): Linear(in_features=288, out_features=128, bias=True)
  (relu2): ReLU()
  (dense2): Linear(in_features=128, out_features=10, bias=True)
))
==================================================
('dense_block.dense1', Linear(in_features=288, out_features=128, bias=True))
==================================================
('dense_block.relu2', ReLU())
==================================================
('dense_block.dense2', Linear(in_features=128, out_features=10, bias=True))
==================================================
​

summary ：

（1）model.modules() and model.named_modules() Method returns an iterator iterator;

（2）model Of modules() Methods and named_modules() Methods will integrate all the components of the whole model （ Including the packaging layer 、 Separate layers 、 Custom layer, etc ） Traverse from shallow to deep , It's just modules() Each element returned is the directly returned layer object itself , and named_modules() Each element returned is a tuple , The first element is the name , The second element is the layer object itself .

（3） How to understand children and modules This difference between . Be careful pytorch Whether it's a model 、 layer 、 Activation function 、 The loss function can be regarded as Module Development of , therefore modules and named_modules Will iterate layer by layer , from the shallower to the deeper , Put each custom block block、 then block Every layer inside is regarded as module To iterate . and children It's more intuitive , It means the so-called “ children ”, So there is no layer by layer iteration .

3、 ... and 、 Model container Containers

In the process of building the model , Another very important concept is containers, Let's take a look first containers Overall framework ：

1、nn.Sequential

nn.Sequential yes nn.Module In the container , Used to wrap a set of network layers in order , It has been explained in detail in the above nn.Sequential Usage of , below debug Take a look at the specific implementation process

class LeNetSequential(nn.Module):
    def __init__(self, classes):
        super(LeNetSequential, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 6, 5),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(6, 16, 5),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),)

        self.classifier = nn.Sequential(
            nn.Linear(16*5*5, 120),
            nn.ReLU(),
            nn.Linear(120, 84),
            nn.ReLU(),
            nn.Linear(84, classes),)

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size()[0], -1)
        x = self.classifier(x)
        return x
net = LeNetSequential(classes=2)

Program entry container.py Of documents Sequential Class , stay __init__ You can see in the first step is to determine whether the added layer is an ordered dictionary , And then nn.Sequential The ordered layers in are added to module in , The construction of the model is completed here , continue debug, It can be seen that the splicing of the model is in container.py Medium forward() To realize .

summary ：
Use nn.Sequential When wrapping a set of network layers in order ：
● Sequence ： The networks at all levels are built in strict order ;
● Bring their own forward()： Bring their own forward in , adopt for The loop executes the forward propagation process in turn .

2、nn.ModuleList

nn.ModuleList yes nn.module The container of , Used to package a set of network layers , Call the network layer iteratively , The main method ：

• append(): stay ModuleList Add the network layer after ;
• extend(): Splice two ModuleList;
• insert(): Specified in the ModuleList Insert network layer in middle position

Be similar to python Medium list operation , But the element is replaced by the network layer , Examples are as follows ：
Use ModuleList To implement a loop iteration 20 The construction of a full connectivity layer network .

class ModuleList(nn.Module):
    def __init__(self):
        super(ModuleList, self).__init__()
        self.linears = nn.ModuleList([nn.Linear(10, 10) for i in range(20)])

    def forward(self, x):
        for i, linear in enumerate(self.linears):
            x = linear(x)
        return x
model = ModuleList()
print(model)
#  Output results 
ModuleList(
  (linears): ModuleList(
    (0): Linear(in_features=10, out_features=10, bias=True)
    (1): Linear(in_features=10, out_features=10, bias=True)
    (2): Linear(in_features=10, out_features=10, bias=True)
    (3): Linear(in_features=10, out_features=10, bias=True)
    (4): Linear(in_features=10, out_features=10, bias=True)
    (5): Linear(in_features=10, out_features=10, bias=True)
    (6): Linear(in_features=10, out_features=10, bias=True)
    (7): Linear(in_features=10, out_features=10, bias=True)
    (8): Linear(in_features=10, out_features=10, bias=True)
    (9): Linear(in_features=10, out_features=10, bias=True)
    (10): Linear(in_features=10, out_features=10, bias=True)
    (11): Linear(in_features=10, out_features=10, bias=True)
    (12): Linear(in_features=10, out_features=10, bias=True)
    (13): Linear(in_features=10, out_features=10, bias=True)
    (14): Linear(in_features=10, out_features=10, bias=True)
    (15): Linear(in_features=10, out_features=10, bias=True)
    (16): Linear(in_features=10, out_features=10, bias=True)
    (17): Linear(in_features=10, out_features=10, bias=True)
    (18): Linear(in_features=10, out_features=10, bias=True)
    (19): Linear(in_features=10, out_features=10, bias=True)
  )
)

nn.ModuleList The specific implementation process of

3、nn.ModuleDict

nn.ModuleDict yes nn.Module The container of , Used to package a set of network layers , Call the network layer by index , The main methods are ：

• clear()： Empty ModuleDict;
• items()： Returns an iteratable key value pair ;
• keys()： Return to the dictionary key ;
• values()： Returns the dictionary value ;
• pop()： Returns a pair of key values , And delete it from the dictionary .

class ModuleDict(nn.Module):
    def __init__(self):
        super(ModuleDict, self).__init__()
        self.choices = nn.ModuleDict({
    
            'conv': nn.Conv2d(10, 10, 3),
            'pool': nn.MaxPool2d(3)
        })

        self.activations = nn.ModuleDict({
    
            'relu': nn.ReLU(),
            'prelu': nn.PReLU()
        })

    def forward(self, x, choice, act):
        x = self.choices[choice](x)
        x = self.activations[act](x)
        return x

Output results ：

Four 、AlexNet The realization of network

AlexNet：2012 Second place higher than in 10 The accuracy rate of more than% is obtained ImageNet Classification task champion , A new era of convolutional neural networks has been created ,AlexNet The characteristics are as follows ：
● use ReLU： Replace saturation activation function , The relief gradient disappears
● use LRN(Local Response Normalization)： Normalize the data , The relief gradient disappears
● Dropout： Improve the robustness of the full connectivity layer , Increase the generalization ability of the network
● Data Augmentation：TenCrop, Color modification
The network structure diagram is as follows ：

Implementation process :

# pytorch in AlexNet The implementation of the 

alexnet = torchvision.models.AlexNet()

#  Implementation process 
class AlexNet(nn.Module):

    def __init__(self, num_classes=1000):
        super(AlexNet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(64, 192, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.avgpool = nn.AdaptiveAvgPool2d((6, 6))
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(256 * 6 * 6, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),
        )

    def forward(self, x):
        x = self.features(x)
        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x