How to use the pre training language model

How to use the pre training model

One 、 Ideas

First of all, consider Data volume of the target model And Correlation between target data and source data .
Generally, it should be based on the different similarity between the data set and the pre training model data set , Adopt different treatment methods .

Above picture
1、 The data set is small , High data similarity
Ideal situation , Sure Use the pre training model as a feature extractor , So it is sometimes called feature extraction .
practice ： Remove the output layer , Treat the rest of the network as a fixed feature extractor , Apply to new datasets .

2、 Data aggregation , High data similarity
frozen In the pretreatment model A few lower layers , Modify the classifier , Then start training again on the basis of the new data set .

3、 The data set is small , Data similarity is not high
frozen In the pre training model Less high-level network , Then retrain the back network , Modify the classifier . The similarity is not high ,so The process of retraining is critical ！！
The data set size is insufficient, which is passed frozen Some of the pre training models Lower network layer Make up for .

4、 Data aggregation , Data similarity is not great
Large data sets ,NN The training process is more efficient . But when the similarity is not high , The pre training model will be very inefficient ,to do： In the pre training model Weights are all initialized Then start training again on the basis of the new data set .

【 notes 】 In the specific operation , Often try many methods at the same time , Choose the best .

Two 、 Get pre training model

1、PyTorch The toolkit torchvision Medium models modular （torchvision.models）, You need to set pretrained=True.
2、tensorflow.keras.application or Can be in TensorFlowHub Website （https://tfhub.dev/google/） Upload and download .
3、huggingFace-transformers（NLP Pre training model library ）