How is the combination of convolution and transformer optimal?

Recent research shows that ,Transformer  Strong remote relationship modeling capability , However, there is nothing we can do to capture high-frequency local information . To solve this problem , This paper proposes  Inception Transformer, abbreviation  iFormer, It can effectively learn the comprehensive features of visual data containing high-frequency and low-frequency information .

say concretely , This paper designs a  Inception mixer take Convolution and Maximum pooling Advantages of migrating to  Transformer  Capture high frequency information in . With the latest mixer Different ,Inception mixer Higher efficiency is achieved through the channel splitting mechanism , At the same time, parallel convolution / The maximum pool path and the self attention path serve as high-frequency mixer and low-frequency mixer It can flexibly model the discrimination information scattered in it .

in consideration of Low-level Layer Play more role in capturing high-frequency details , and High-level Layer Play more role in modeling low-frequency global information , The author further introduces frequency ramp structure, That is, gradually reduce to high-frequency mixer Dimensions , And increase low-frequency mixer Dimensions （ A word is ResNet Hierarchical design idea of ）, The high-frequency and low-frequency components of different layers can be effectively weighed .

On a series of visual tasks  iFormer  Benchmarking , And shows its application in image classification 、COCO Detection and ADE20K Excellent performance in segmentation . for example ,iFormer-S  stay ImageNet-1K Up to 83.4% Of top-1 Accuracy rate , Than  DeiT-S  Higher than 3.6%, The only 1/4 Parameters and 1/3 Of FLOPs Even slightly better than the larger model  Swin-B (83.3%).

Thesis link ：

https://arxiv.org/abs/2205.12956