442 authors, 100 pages! It took Google 2 years to release the new benchmark big bench | open source

White cross From the Aofei temple
qubits | official account QbitAI

A piece of AI The paper ,442 An author .

There is also a chapter devoted to the author's contribution .

100 More than half of the pages are references ……

No , It's popular now This kind of paper Do you ？

see , Google's latest paper ——Beyond The Imitation Game: Quantifying And Extrapolating The Capabilities Of Language Models.

So the author column becomes like this ……

come from 132 A research scholar at an institution , It took two years to propose a new benchmark for large language model BIG-bench.

On this basis, the OpenAI Of GPT Model ,Google-internal dense transformer Architecture, etc , Model scale horizontal 6 An order of magnitude .

Final results showed , Although the model performance improves with the expansion of the scale , But it is far from the human performance .

For this work ,Jeff Dean Forward likes ：Great Work.

New benchmark for big language model

What did Lai Kangkang say in this paper .

With the expansion of the scale , The performance and quality of the model have been improved , There may also be some transformative effects , But these performances have not been well described before .

Some existing benchmarks have certain limitations , The scope of assessment is narrow , Performance scores quickly reach saturation .

such as SuperGLUE, After the introduction of the benchmark 18 months , The model implements “ Beyond the human level ” Performance of .

Based on this background ,BIG-bench It was born .

Currently it is controlled by 204 A task consists of , The content covers linguistics 、 Child development 、 mathematics 、 Commonsense reasoning 、 biology 、 physics 、 Social prejudice 、 Problems in software development, etc .

There is also a panel of human experts , Also performed all tasks , To provide baseline levels .

For the convenience of more organizations , The researchers also gave BIG-bench Lite, A small but representative subset of tasks , Easy and faster assessment .

And open source implementation benchmarks API Code for , Support Task Evaluation on publicly available models , And the lightweight creation of new tasks .

The final assessment results can be seen , The scale spans six orders of magnitude ,BIG-bench The overall performance of the model increases with the scale of the model 、 The number of training samples increases .

But compared with human baseline level , Still perform poorly .

Specific tasks , The performance of the model will steadily improve with the increase of the scale . But sometimes , There will be sudden breakthrough performance on a specific scale .

Besides , It can also assess the social bias of the model .

Besides , They were also surprised to find that the model was OK get Some hidden skills . such as , How to move regularly in chess .

The author contributed 14 page

It is worth mentioning that , Maybe because there are too many authors , At the end of the paper, there is a chapter devoted to the author's contribution .

I wrote with great ease 14 page , This includes core contributors 、Review Of 、 To provide a task ……

The rest , also 50 Page references .

Okay , Interested friends can poke the following link to Kangkang's paper .

Thesis link ：
https://arxiv.org/abs/2206.04615
GitHub link ：
https://github.com/google/BIG-bench
Reference link ：
https://twitter.com/jaschasd/status/1535055886913220608