Understanding of inverted residuals

Analysis of inverse residuals and linear bottlenecks - MobileNetV2_ Xiaoyezhu's blog -CSDN Blog _ Inverse residual structure  

First of all, to understand the inverted residual, we must first understand the form of residual

Residual structure ：

1. use 1*1 Convolution dimensionality reduction , For example, the input is 256, Dimensionality reduction 64

2. Using convolution kernel as 3*3 form

3. use 1*1 Convolution dimension ., such as 64 become 256

After understanding the residual structure , Now start to learn the inverted residual structure

1. use 1*1 Convolution dimension , For example, the input is 64, Dimensionality reduction 256

2. The convolution kernel is deeply separable 3*3 form

3. use 1*1 Convolution Dimension reduction . such as 256, Dimensionality reduction 64

The activation function here adopts relu6

# _*_coding:utf-8_*_
import torch
import torch.nn as nn


class InvertedResidualsBlock(nn.Module):
    def __init__(self, in_channels, out_channels, expansion, stride):
        super(InvertedResidualsBlock, self).__init__()
        channels = expansion * in_channels
        self.stride = stride

        self.basic_block = nn.Sequential(
            nn.Conv2d(in_channels, channels, kernel_size=1, stride=1, bias=False),
            nn.BatchNorm2d(channels),
            nn.ReLU6(inplace=True),
            nn.Conv2d(channels, channels, kernel_size=3, stride=stride, padding=1, groups=channels, bias=False),
            nn.BatchNorm2d(channels),
            nn.ReLU6(inplace=True),
            nn.Conv2d(channels, out_channels, kernel_size=1, stride=1, bias=False),
            nn.BatchNorm2d(out_channels)
        )
        # The shortcut operation does not affect the number of channels
        self.shortcut = nn.Sequential(
            nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=1, bias=False),
            nn.BatchNorm2d(out_channels)
        )

    def forward(self, x):
        out = self.basic_block(x)
        if self.stride == 1:
            print("With shortcut!")
            out = out + self.shortcut(x)
        else:
            print("No shortcut!")
        print(out.size())

        return out


if __name__ == "__main__":
    x = torch.randn(16, 3, 32, 32)
    # no shortcut
    net1 = InvertedResidualsBlock(3, 6, 6, 2)
    # with shortcut
    net2 = InvertedResidualsBlock(3, 6, 6, 1)
    y1, y2 = net1(x), net2(x)