A survey on dynamic neural networks for natural language processing, University of California

author ：Canwen Xu, Julian McAuley

brief introduction ： Scale large objects effectively Transformer  Models are the main driving force behind the latest advances in natural language processing . Dynamic neural network is a new research direction , It can dynamically adjust the calculation path of neural network according to the input , Thus, the calculation amount and time can be increased in a sub linear way . Dynamic neural networks may be a promising solution , It can solve the increasing number of parameters of the pre training language model , It can use trillions of parameters for model pre training , Faster reasoning on mobile devices . In this review , The author summarizes the progress of three dynamic neural networks in natural language processing ： skimming （skimming）、 Hybrid expert model （mixture of experts） And early exit reasoning （early exit）. The author also emphasizes the current challenges and future research direction of dynamic neural network .

 

Paper download ：https://arxiv.org/pdf/2202.07101.pdf