ICML 2022 | flowformer: task generic linear complexity transformer

Paper title ：Flowformer: Linearizing Transformers with Conservation Flows

Thesis link ：https://arxiv.org/pdf/2202.06258.pdf

Code link ：https://github.com/thuml/Flowformer

This paper deeply studies the quadratic complexity of attention mechanism , By introducing the conservation principle in network flow into the design , Naturally, competition mechanism is introduced into attention Computing , It effectively avoids ordinary attention problems .

What we proposed Task common The backbone network Flowformer, Realized Linear complexity , At the same time Long sequence 、 Vision 、 Natural language 、 The time series 、 Reinforcement learning Achieved excellent results in five major tasks .

In the application of long sequence modeling , Such as protein structure prediction 、 Long text understanding, etc ,Flowformer It has good application potential . Besides ,Flowformer in “ No special inductive preference ” The design concept of is also of good enlightening significance to the research of general infrastructure .

Flow-Attention The pseudo-code is as follows ：

Main experimental results ：