Yalm 100b: 100billion parameter open source large model from yandex, Russia, allowing commercial use

GitHub Address ： https://github.com/yandex/YaLM-100B （ Just released a few days , There has been a 2400 It's a star ）

Yandex Russian search giant , The official blog This is how it is introduced ：

Over the past year , We have been Alice Voice assistant and Yandex Use... In search YaLM Serial language model . today , We will open source the largest YaLM Model , Yes 1000 One hundred million parameters . We spent it. 65 Days of time in 800 individual A100 On the graphics card and 1.7 TB Online text 、 Training this model on books and countless other resources . We are GitHub Models and useful materials are published on , use Apache 2.0 The license , Allow for research and commercial use . It is the largest freely available English in the world GPT Neural network like .

The blog also conscientiously introduces many experiences of model training acceleration , Including how to find bottlenecks 、 Use quick data types 、 Speed up GPU The operating 、 Reduce memory access 、 Ban Dropout、 signal communication 、ZeRO Optimizer, etc , Recommended reading .