VAN(DWConv+DWDilationConv+PWConv)

1. introduction

Initial self attention mechanism （self-attention） My proposal is in NLP field , But because of its ability of global feature extraction , Self attention mechanism soon swept CV field . But for 2D Image ,self-attention There are the following deficiencies ：（1） Lay the image 1D The sequence is lost 2D structural information .（2） Square complexity makes high-resolution pictures （ Such as 800*800） It's too expensive to compute .（3） Only the spatial adaptability is extracted , And ignore the channel channel Adaptability of dimensions . therefore , be based on LKA（Large Kernel Attention） A new network architecture is proposed VAN. Although the architecture is relatively simple , Namely DWConv+DWDilationConv+PWConv The superposition , But its performance can match the current SOFT Vision of Transformer as well as CNN Comparable . As a general backbone network , In image recognition 、 object detection 、 Good results have been achieved in downstream tasks such as semantic segmentation and individual segmentation .

2. Implementation method

There are two main ways to realize the attention module , One is Transformer Medium self attention, But for 2D There are three kinds of defects in images ; The second is to use large kernel convolution （ Be similar to MobileNet_V3 Used in SE modular ）, But there are too many parameters , Too much computing overhead . Therefore, the idea of decomposing the large core attention module is proposed .

2.1 Big core attention module LKA

 

Decompose the large convolution kernel into DWConv+DWDilationConv+PWConv In the form of ,

（1）DWConv Extract local detailed features

（2）DWDilationConv Extract a wide range of features

（3）PWConv Extract channel C Dimensional characteristics

After the above three stages , Calculate the weight of the corresponding pixel .

3. The overall architecture

Calculate the parameters ：

Ordinary convolution ： Convolution kernel H* Convolution kernel W* Enter the number of channels * Number of output channels

DW Convolution ： Convolution kernel H* Convolution kernel W * Number of output channels

 

  By the table 5 It can be seen that ,d=3,K=21 Precision basis This is saturated and the calculation parameters are not too many .

  By the table 1 It can be seen that ,VAN All in all Local features 、 A wide range of features 、 Spatial adaptability and adaptability of channel dimension .

 VAN from 4 individual stage form . Every stage from LKA Module and feedforward neural network CFF form , Overall architecture and transformer Of self-attention+ Two layers of MLP The architecture of is very similar .（ stay ConNeXt Mentioned in ,transformer Good performance is inseparable from the design of its architecture ）

And swin-transformer Agreement , Every stage first block Take down sampling down sample, Module image resolution resolution All reduced , And the number of channels channels increase . By controlling the slip step of convolution stride To control the sampling rate . Except for the first one block Lower sampling outside , rest block The input and output characteristic diagram of is consistent .

4. result

4.1 Test details

stay ImageNet Test on , Data enhancement ,random clipping, random horizontal flipping, label-smoothing [59], mixup [102],cutmix [100] and random erasing [105] . Use AdamW Optimizer trained 310 individual epochs,momentum=0.9, weight decay=5 × 10−2. Initial learning rate LR=5 × 10−4, Use the descending cosine cosin Learning rate ,Layer-Scale technology ,batch_size=1024,Exponential moving average (EMA) To improve the training process .

4.2 stay ImageNet The experimental results

  By the table 6 You know ,VAN stay top1 The best accuracy is achieved on both sides , This is the structure that the author calls out according to intuition , I believe that after optimizing some details of the architecture parameters , It must be able to improve . because VAN A combination of VIT and CNN, Use attention to synthesize global information , The local details are handled carefully .

Ablation Experiment

DWConv+DWDilationConv+PWConv Be short of one cannot ,

The lack of DWConv, The accuracy is down 0.5%;

The lack of DWDilationConv, The accuracy is down 1.3%,

Lack of attention mechanism , The accuracy is down 1.1%;

The lack of PWConv, The accuracy is down 0.8%.

5. Visual analysis

Use Grad-CAM Make visual discovery , It is superior to SWIN_T as well as ConvNeXt.

6. Future outlook

（1） Just intuitively propose the framework , Architecture parameters can also be optimized .

In this paper, we only demonstrate an intuitive structure. There are a lot of potential improvements such as adopting different kernel size, introducing multi-scale structure and using multi-branch structure.

（2） Large scale self supervised learning and transfer learning .

A combination of ViT And CNN Advantages of Architecture , It can be extracted 2D The characteristics of the structure and the output can be dynamically adjusted according to the input .

we believe VAN can achieve better performance in image self-supervised learning and transfer learning field.