Started a natural language model bloom

stay 2 Years ago ,OpenAI Made a 1750 A neural network model with 100 million parameters , namely GPT-3( Is its predecessor GPT-2, about 15 One hundred million parameters , Of 100 Many times ). It is a pity , GPT-3 There is no open source data set for its pre trained parameters , Users can only invoke what they provide API, Because of this API You must apply for , So I haven't had a chance to use . however , There are already many companies based on GPT-3 Developed some applications , for instance GitHub Just open one GitHub Copilot For intelligent completion code , If you are interested, you can try .

generally ,GPT-3 The neural network model at this level has only OpenAI, Giant technology companies like Google can develop and train , After all 1750 The number of arguments is not a small number . however , One by about 1000 International volunteers composed of many academic volunteers do not believe in evil , They are using value 700 Million dollars of public funds to sponsor the calculation of time to train a person with 1760 A natural language model with 100 million parameters ,BLOOM.

Out of interest , I'm in my own Macbook Pro Tested this on BLOOM-1b3 Model , As the name suggests, the model has 13 Million parameters . Why not test the complete 1750 Billion parameters ？ Because light reading 13 100 million parameters to Python in , You need to occupy 6G Left and right memory , complete 1750 One hundred million parameters , At least one 1T Memory server . And the complete parameter version has not been completed yet , Still training .

BLOOM Is based on Transformers, and Transformers The installation requirements of meet the following three points

• Python 3.6+
• Flax 0.3.2+/ PyTorch 1.3.1+ / TensorFlow 2.3+ Any frame
• Rust( Used to compile tokenizers)

stay Mac On , We can go through homebrew To install rust

brew install rust

In the framework of deep learning , I chose PyTorch, Because it supports the use of M1 Chip GPU Accelerate

#  Installing a virtual environment 
python3 -m pip install --user --upgrade pip
python3 -m pip install --user virtualenv
#  Creating a virtual environment 
python3 -m venv pytorch
source pytorch/bin/activate
#  install pytorch
pip3 install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu
#  install transformers
pip3 install transformers

After open the python, Load our model ( First run , transformers Will automatically help download the model )

from transformers import AutoTokenizer, AutoModelForCausalLM

#  participle 
tokenizer = AutoTokenizer.from_pretrained('bigscience/bloom-1b3')
#  Model 
model = AutoModelForCausalLM.from_pretrained('bigscience/bloom-1b3')

Because I know nothing about natural semantic processing , So only according to the sample code , Use transformres Of pipeline, Build a text generation tool .

from transformers import pipeline
generator = pipeline(task="text-generation", model=model, tokenizer=tokenizer)

#  English input 
generator("bioinformatics is ", max_length=30, num_return_sequence=5)

#  Output results 
[{'generated_text': 'bioinformatics is  a branch of computer science that deals with the analysis of data and the design of algorithms that can be used to solve problems in'}]

#  Chinese input 
generator(" Bioinformatics is a subject ")
#  Output results 
[{'generated_text': ' Bioinformatics is a new interdisciplinary subject , It involves bioinformatics 、 Computer science 、 mathematics '}]

From the results ,13 Billion parameter BLOOM The performance of the model has been very much in line with my expectations , Far more than before Transformers Default when testing GPT-2 Model .

Unfortunately, my ability is limited , Yes BLOOM Our exploration stops here , Wait until I am right Transformers With more mastery , Update the relevant content .

Reference material

• https://www.nature.com/articles/d41586-022-01705-z