Technical practice behind bloom model: how to refine 176billion parameter model?

In recent years , Training larger language models has become the norm . Although these models have not been published for further study, problems are often discussed , But the hidden knowledge about how to train these models has received little attention . This paper aims at 176B Parametric language model BLOOM The example of illustrates how to train the technology and engineering technology behind such models in hardware and software to change this .

But first of all , We want to thank the company 、 Key people and groups , They let a small group of dedicated people train one 1760 The amazing feat of the billion parameter model becomes possible .

Then we will discuss the hardware settings and main technical components .