【ARXIV2205】Inception Transformer

【ARXIV2205】Inception Transformer

The paper ：https://arxiv.org/abs/2205.12956

Code ：https://github.com/sail-sg/iFormer

1、 Research motivation

The core idea of this paper is still ： hold attention and CNN Combination （ Google's Inception）, But the starting point is different . The author gives VIT Of Fourier spectrum , It is found that the energy is basically concentrated in the low-frequency part , For the edges in the image 、 Insufficient extraction of high-frequency information such as corners .（ It's easy to understand ,attention Essentially, it is similar to the global weighted sum , There is a loss of local information ）

2、 Inception mixer

The main contribution of this paper is to improve attention , A new module is proposed ： Inception mixer. The author's idea is very direct , As shown in the figure below , In the existing VIT Add high-frequency branches to the structure ！

（1） High frequency branch . It can be seen that , The high-frequency branch comes from the classic Inception（ As shown in the figure below ）, Among them linear The essence of layer is 1x1 Convolution .

The characteristics of the input are $C$ Channels , Divide it into $C_h$ and $C_l$ Channels , It is used to extract high-frequency and low-frequency features respectively . For high frequency characteristics , Evenly divided into two parts $X_{h1}$ and $X_{h2}$（ Are all $C_h/2$ Channels ）, Do the following ：

$$Y_{h1}=\text{FC}(\text{MaxPool}(X_{h1}))$$

$$Y_{h2}=\text{DwConv}(\text{FC}(X_{h2}))$$

（2） Low frequency branch . The low frequency branch is traditional MHSA, Because other branches bring extra computation , So this branch goes first average pooling operation , Then enter MHSA Calculate later upsample operation .

Last , The results of high frequency and low frequency are directly spliced together ：$Y_c=\text{Concat}(Y_l,Y_{h1},Y_{h2})$.

Last , Because the direct interpolation in the low-frequency upsampling operation , Cause adjacency token Too smooth and similar , To solve this problem , The author adds one DwConv, Specific for ：$Y=\text{FC}(Y_c+\text{DwConv}(Y_c))$

3、 Overall framework

The author adopts the current mainstream 4 Stage transformer framework , To build the small, base, large Three models , The details are shown in the table below . As can be seen from the table , In the shallow stage of the network , high frequency （conv） Account for a large proportion , Low frequency （MHSA） Account for a small proportion . In the deep stage of network , It's the other way around . This is convoluted with the current mainstream Transformer The conclusion of the combination method is basically the same . meanwhile , In the conclusion , The author also acknowledges , The ratio between high frequency and low frequency needs to be determined according to experience , It's this method limitation.

This method has achieved very good performance in image classification tasks . The author also did target detection 、 Experiments on semantic segmentation , For details, please refer to the author's paper .