The Impossible Triangle of NLP?

author  |  Prince Changqin  

Arrangement  | NewBeeNLP

Come and have a look today NLP The Impossible Triangle of the model , And based on this, some future research directions .

• Paper： Impossible Triangle: What's Next for Pre-trained Language Models?^[1]

PLM The Impossible Triangle of means ：

1. Medium model size （1B following ）

2. SOTA few-shot Ability

3. SOTA Fine tuning capability

At present all PLM One or more of them are missing . A lot of knowledge is injected into distillation 、 Data to enhance 、Prompt To mitigate these deficiencies , But it brings new workload in practice . This paper provides a future research direction , Break down the task into several key phases to achieve the Impossible Triangle .

The pre training model is well known , But people have not found in small and medium-sized models few-shot even to the extent that zero-shot The ability of , The big model does , But because it is too big, it is very inconvenient in actual use . But the reality is that many times we don't label too much data , Need this kind of few-shot The ability of .

Impossible Triangle

As shown in the figure below ：

• P1 It is used for efficient deployment with reasonable computing resources

• P2 Used in scenes with no or a small amount of annotation data

• P3 It is used in scenes with relatively large amount of annotation data

A good evidence is Google Recently published PaLM, The paper found that , Model size and few/zero-shot There is a difference between performance Discontinuous The promotion of . for instance , And 8B and 62B Model comparison ,PaLM Of 540B Show breakthrough improvement in many tasks .

For the Impossible Triangle , Actually PLM Can often achieve 1-2 individual ：

• medium-sized PLM（1B following ）：P1+P3

• in super-large scale PLM：P2. It is worth noting that ：zero/few-shot The effect is still not as good as that of supervision ; In addition, most of the fine-tuning is not as good as the medium size PLM The result of fine tuning （ The reason is probably that the model is too big ）.

Current strategy

For model size （ Lack of P1）：

• In general, the super large model shows excellent zero/few-shot Ability and powerful performance after fine tuning occurs .

• The common method is 「 Distillation of knowledge 」.

• There are two questions ： The student model can hardly achieve the effect of the teacher model ; Too many models hinder effective reasoning , Make it inconvenient as a teacher model .

For the poor zero/few-shot Ability （ Lack of P2）：

• This is common in medium models ： It can be achieved through fine adjustment SOTA, but zero/few-shot Relatively inadequate capacity .

• The method is 「 Generate pseudo tags and samples from other models , Or noise injection expansion data 」.

• however , The change of pseudo data quality and the diversity of data types in different tasks pose challenges to universally applicable solutions .

Poor supervised training performance （ Lack of P3）：

• This is typical in the fine-tuning of oversized models , The computing resources are limited or the amount of training data is not enough to fine tune it .

• The typical strategy is 「Prompt Study 」, You can use hard prompts （ Discrete text template ） Or soft tips （ Continuous formwork ）, So that only the parameters of the hard prompt word or the soft prompt are updated during tuning .

• however , The method is right Prompt The selection and training data are particularly sensitive , Still not as good as medium size PLM + Supervised .

Future approaches

This paper presents a multi-stage method .

• Stage 1： The goal is to achieve something （ Impossible in the triangle ） Properties needed , Improve missing attributes . such as ,SOTA A supervised medium model can improve few-shot Learning performance ,SOTA few-shot Large models of capabilities are compressed to smaller models with better supervised performance .

• Stage 2： Implementation of three attributes PLM Developed for a few tasks . You can take advantage of the unique characteristics of the target task , For example, performance is less dependent on the scale of training data ,zero/few-shot And have a supervised performance gap Smaller, etc .

• Stage 3： Based on stage 1 And stages 2, In general NLP Implement three properties on the task . Possible methods include ： Pre train a medium-sized model with a large amount of data , Better knowledge distillation , General data enhancement methods, etc .

Although this article is not long , But the entry point is quite interesting , Mitigation strategies for each attribute are also analyzed ： Distillation of knowledge 、 Data to enhance 、Prompt Learning, etc. , Based on this, the future research direction , In fact, it seems to be a very natural idea . But this impossible triangle is really interesting .

Communicate together

I want to learn and progress with you ！『NewBeeNLP』 At present, many communication groups in different directions have been established （ machine learning / Deep learning / natural language processing / Search recommendations / Figure network / Interview communication /  etc. ）, Quota co., LTD. , Quickly add the wechat below to join the discussion and exchange ！（ Pay attention to it o want Notes Can pass ）

Resources for this article

[1]

Impossible Triangle: What's Next for Pre-trained Language Models?: https://arxiv.org/abs/2204.06130