Shiftvit uses the precision of swing transformer to outperform the speed of RESNET, and discusses that the success of Vit does not lie in attention!

Attention mechanism is widely considered to be Vision Transformer(ViT) The key to success , Because it provides a flexible and powerful way to model spatial relationships . However , The attention mechanism is really ViT An integral part of ？ Can it be replaced by some other alternatives ？ In order to uncover the role of attention mechanism , The author reduces it to a very simple case ：ZERO FLOP and ZERO parameter.

To be specific , The author reexamined Shift operation . It does not contain any parameters or arithmetic calculations . The only operation is to exchange a small number of channels between adjacent features . Based on this simple operation , The author constructs a new Backbone, namely ShiftViT, among ViT The attention layer in is shift Operation replaced .

It's amazing ,ShiftViT Worked well on several mainstream tasks , Such as the classification 、 Detection and segmentation . Performance is even better than Swin Transformer Better . These results suggest that , The attention mechanism may not make ViT The key to success . It can even be replaced by an operation with zero parameters . In the future work , We should pay more attention to ViT The rest of .

Thesis link ：

https://arxiv.org/abs/2201.10801

Shift Block The detailed architecture of is shown in the figure below ：