Model definition pytorch learning

1. PyTorch How the model is defined : nn.Module(Sequential,ModuleList,ModuleDict)

• Module  Class is  torch.nn  modular 里 Provide a model construction class (nn.Module), It's all nerves ⽹ Base class of network module , We can inherit it to define the model we want ;

• PyTorch The model definition should include two main parts ： Initialization of each part （__init__）; Data flow definition （forward）

be based on nn.Module, We can go through Sequential,ModuleList and ModuleDict There are three ways to define PyTorch Model .

1.1 Sequential: nn.Sequential()—— Direct arrangement / Use OrderedDict

When the forward calculation of the model is simple 串 When connecting the calculation of each layer , Sequential  Class can pass through 更 Add a simple way to define the model . It can receive an ordered Dictionary of sub modules (OrderedDict) Or a series of sub modules can be added one by one as parameters  Module  Reality of 例, The forward calculation of the model is to put these real 例 Calculate one by one in the order of addition . We combine Sequential And how it is defined ：

class MySequential(nn.Module):
    from collections import OrderedDict
    def __init__(self, *args):
        super(MySequential, self).__init__()
        if len(args) == 1 and isinstance(args[0], OrderedDict): #  If the incoming is a OrderedDict
            for key, module in args[0].items():
                self.add_module(key, module)  
                # add_module Methods will module Added to the self._modules( One OrderedDict)
        else:  #  What comes in is some Module
            for idx, module in enumerate(args):
                self.add_module(str(idx), module)
    def forward(self, input):
        # self._modules Return to one  OrderedDict, Ensure that it will traverse into... In the order in which the members are added 
        for module in self._modules.values():
            input = module(input)
        return input

Use Sequential To define the model , Just arrange the layers of the model in order , Depending on the layer name , There are two ways to arrange ：

        1. Direct arrangement

import torch.nn as nn
net = nn.Sequential(
        nn.Linear(784, 256),
        nn.ReLU(),
        nn.Linear(256, 10), 
        )
print(net)
'''
Sequential(
  (0): Linear(in_features=784, out_features=256, bias=True)
  (1): ReLU()
  (2): Linear(in_features=256, out_features=10, bias=True)
)
'''

        2. Use OrderedDict

import collections
import torch.nn as nn
net2 = nn.Sequential(collections.OrderedDict([
          ('fc1', nn.Linear(784, 256)),
          ('relu1', nn.ReLU()),
          ('fc2', nn.Linear(256, 10))
          ]))
print(net2)
'''
Sequential(
  (fc1): Linear(in_features=784, out_features=256, bias=True)
  (relu1): ReLU()
  (fc2): Linear(in_features=256, out_features=10, bias=True)
)
'''

You can see , Use Sequential The benefit of defining a model is simplicity 、 Easy to read , meanwhile Use Sequential The defined model does not need to be written forward, Because the order has been defined . But use Sequential It will also make the model definition inactive , For example, when you need to add an external input in the middle of the model, it is not suitable to use Sequential The way to achieve . It needs to be selected according to the actual needs .

1.2 ModuleList：nn.ModuleList() Store different modules

ModuleList  Receive a sub module （ Or layer , Need to belong to nn.Module class ） Of 列 Table as input , Then it can be similar List Do it that way append and extend operation . meanwhile , The weights of sub modules or layers will also be automatically added to the network .

net = nn.ModuleList([nn.Linear(784, 256), nn.ReLU()])
net.append(nn.Linear(256, 10)) # #  similar List Of append operation 
print(net[-1])  #  similar List Index access for 
print(net)
'''
Linear(in_features=256, out_features=10, bias=True)
ModuleList(
  (0): Linear(in_features=784, out_features=256, bias=True)
  (1): ReLU()
  (2): Linear(in_features=256, out_features=10, bias=True)
)
'''

Pay special attention to ,nn.ModuleList  There is no definition of a network , It just stores different modules together .ModuleList The order of elements in the network does not represent their real position order in the network , Need to go through forward The function specifies the order of each layer before completing the definition of the model . For specific implementation, use for The loop is done ：

class model(nn.Module):
  def __init__(self, ...):
    super().__init__()
    self.modulelist = ...
    ...
    
  def forward(self, x):
    for layer in self.modulelist:
      x = layer(x)
    return x

1.3 ModuleDict：nn.ModuleDict() Easy to add names

ModuleDict and ModuleList Has a similar effect , It's just ModuleDict It is more convenient to add names to the layers of neural networks .

net = nn.ModuleDict({
    'linear': nn.Linear(784, 256),
    'act': nn.ReLU(),
})
net['output'] = nn.Linear(256, 10) #  add to 
print(net['linear']) #  visit 
print(net.output)
print(net)
'''
Linear(in_features=784, out_features=256, bias=True)
Linear(in_features=256, out_features=10, bias=True)
ModuleDict(
  (act): ReLU()
  (linear): Linear(in_features=784, out_features=256, bias=True)
  (output): Linear(in_features=256, out_features=10, bias=True)
)
'''

1.4  Comparison and application scenarios of the three methods

Sequential Suitable for rapid verification of results , Because it's clear which layers to use , Just write it directly , You don't need to write at the same time __init__ and forward;

ModuleList and ModuleDict When an identical layer needs to appear multiple times , It is very convenient to realize , Sure ” One line at most “;

When we need the information of the previous layer , such as ResNets Residual calculation in , The results of the current layer need to be fused with those of the previous layer , In general use ModuleList/ModuleDict It's more convenient .

2.  Use model blocks to quickly build complex networks

In the previous section, we introduced how to define PyTorch Model of , The examples given are all in torch.nn To complete the . This definition is easy to understand , In the actual scene, it is not necessarily conducive to the use of . When the depth of the model is very large , Use Sequential Defining the model structure requires adding hundreds of lines of code to it , Very inconvenient to use .

For most model structures （ such as ResNet、DenseNet etc. ）, If we look closely, we will find , Although the model has many layers , But there are many recurring structures . Considering that each layer has its input and output , Several layers are connected in series ” modular “ It also has its inputs and outputs , If we can define these recurring layers as a ” modular “, Each time, you only need to add corresponding modules to the network to build the model , This will greatly facilitate the process of model building .

3. PyTorch Modify the model

4. PyTorch Model saving and reading

Reference

PyTorch Medium ModuleList and Sequential: Distinguish and use scenarios - You know

5.1 PyTorch How the model is defined — Explain profound theories in simple language PyTorch