Depthwiseseparableconvolution: depthwise convolution and pointwise convolution

0、 Preface

Deep separable convolution needless to say , Lightweight network architecture is an inextricable topic , As long as the depth of contact learning more or less will contact .

Deep separable convolution is Depthwise Separable Convolution, This convolution divides a conventional convolution process into two complete ：Depthwise Convolution sum Pointwise Convolution , When ensuring the same output , The amount of calculation is greatly reduced .

First of all to see , How to realize a conventional convolution . When I was there pytorch Define a convolution , This convolution accepts one 3 passageway tensor, Output one 4 passageway tensor：

nn.Conv2d(in_channels=3,
          out_channels=4,
          kernel_size=3)

Its schematic diagram ：
Let's calculate the parameter quantity ：parameters=3*3*3*4=108
In terms of promotion , The parameter is ：parameters=kernel_size*kernel_size*in_channels*out_channels

If we look at it from the perspective of channel number transformation , Convolution is to put 3 The tunnel tensor Change into 4 The tunnel tensor, Let's see how deep separable convolution realizes channel change .

1、 Depth separates the convolution

Deep separable convolution is actually two steps ：depthwise Convolution +pointwise Convolution

We know point convolution （pointwise） A major role of is to expand channels and compress channels , In fact, we can directly in 3 The tunnel tensor On , Directly use point convolution to raise the dimension , take 3 The passage becomes 4 passageway , But doing so violates the essence of convolution .

In essence, convolution is to extract features at a uniform speed through convolution , Different features correspond to different channels , So the number of channels is changed . So we want to replace ordinary convolution with deep separable convolution , Convolution operation is also required , Then change the number of channels .

So the first stage depthwise Convolution is the operation of convolution , Second stage pointwise Convolution is changing the number of channels , Make it conform to the predefined number of output channels .

The following two pictures are depthwise Convolution sum pointwise Schematic diagram of convolution .

1.1 depthwise Convolution

The most difficult thing to understand about deep separable convolution should be depthwise Convoluted , It may be very simple at first glance , But if you use pytorch Implementation is troublesome , Because it involves the concept of group convolution .

The concept of group convolution can be seen Grouping convolution （Group Converlution）, actually depthwise Convolution is the number of groups = Enter the number of channels = Special grouping convolution of the number of output channels , So you can see that depthwise Convolution is just a separate convolution operation for each channel , But there is no cross channel calculation , In this way, perfect feature extraction cannot be carried out , and depthwise Convolution also has no ability to change the number of channels

1.2 pointwise Convolution

in the light of 1.1 Problems in , Use pointwise Convolution can be achieved .pointwise It is essentially a general convolution , It's just kernel_size=1 nothing more . To be honest, there's nothing to say .

2、 Code implementation

import torch
from torch import nn



class DSC(nn.Module):
    """  Depth separates the convolution ：https://zhuanlan.zhihu.com/p/80041030 https://zhuanlan.zhihu.com/p/490685194  First, depthwiseConv, In essence, it is grouping convolution , In deep separable convolution , The number of groups of group convolution = Enter the number of channels = Number of output channels , The number of channels in this part remains unchanged   Then the pointwisejConv, Point convolution , This part is responsible for expanding the number of channels , So its kernel_size=1, no need padding """
    def __init__(self, in_channel, out_channel, ksize=3,padding=1,bais=True):
        super(DSC, self).__init__()

        self.depthwiseConv = nn.Conv2d(in_channels=in_channel,
                                       out_channels=in_channel,
                                       groups=in_channel,
                                       kernel_size=ksize,
                                       padding=padding,
                                       bias=bais)
        self.pointwiseConv = nn.Conv2d(in_channels=in_channel,
                                       out_channels=out_channel,
                                       kernel_size=1,
                                       padding=0,
                                       bias=bais)

    def forward(self, x):
        out = self.depthwiseConv(x)
        out = self.pointwiseConv(out)
        return out

if __name__=="__main__":
    from torchsummary import summary
    dsc=DSC(in_channel=3,out_channel=8,ksize=3,padding=1,bais=False).cuda()
    summary(dsc,input_size=(3,48,48))

The parameter is 51, If it's an ordinary convolution Conv2d(3,8,3), The parameter is 216（ No addition bias）, It can be seen that the parameter quantity decreases greatly .

Reference resources

Depthwise Convolution and Pointwise Convolution