Naacl 2022 | TAMT: search the transportable Bert subnet through downstream task independent mask training

Paper title ：Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training

Thesis link ：https://arxiv.org/abs/2204.11218

Code link ：https://github.com/llyx97/TAMT

With BERT Marked by the advent of ,“ Preliminary training + Downstream task fine tuning ” Become NLP A mainstream paradigm of tasks . Research trends show , The increasing parameter quantity is the pre training language model （PLM） A key factor in success . However , This also increases PLM The cost of fine tuning and deployment of downstream tasks . meanwhile , Recent assumptions about lottery （Lottery Ticket Hypothesis, LTH [1]） The study of  [2] Find out , Weight based pruning （magnitude-based pruning） Can be in pre training BERT Found some in the model winning ticket Subnetwork , They can be migrated to multiple downstream tasks to fine tune , And obtain and complete BERT Quite effective .

Through preliminary experimental analysis , We found that these winning ticket The sub network has a good downstream task migration effect , And their effects on pre training tasks . Pictured 1 Shown , stay 50% sparsity Next ,oneshot magnitude pruning （OMP）[1]  The resulting sub network is pre trained mask language modeling（MLM） Both tasks and downstream tasks surpass the random subnetwork obviously . However , With sparsity [2]  Keep going up ,OMP In the subnetwork MLM The effect on tasks and downstream tasks decreases at the same time . This means that if we can find better performance in pre training tasks BERT Subnetwork , Then they are likely to have better migration capabilities for downstream tasks .

chart 1 OMP Subnetworks and random subnetworks （Rand） Downstream missions （ Left ） And pre training tasks （ Right ） The performance of the

chart 2 TAMT On pre training tasks （MLM Or knowledge distillation ） Learning sub network structure , Then migrate it to different downstream tasks for fine-tuning

Based on the above motives , We propose downstream task independent mask training （Task-Agnostic Mask Training,TAMT） Method . Pictured 2 Shown ,TAMT Optimize pre training tasks BERT The structure of the subnetwork （ Do not change the pre training parameter value ）, So the sub network has better performance in the pre training task . Subsequently, the searched sub network will be migrated to a variety of downstream tasks for fine-tuning training .