Bert bidirectional encoder based on transformer

BERT（Bidirection Ecoder Representations from Transformers）： Left and right information , Deep two-way Transformer code （enceder） Model

Feature based model feature-based：ELMo, be based on RNN

Model based on fine tuning fine-tuning：GPT, One way language model

Catalog

1.MLM:Mask Language Model Mask language model （ Cloze form ） Self supervision

2.BERT frame ：

3. Loss function

 4. summary

---

1.MLM:Mask Language Model Mask language model （ Cloze form ） Self supervision

for example i am a little boy,she is a beautiful gilr. Convert into MLM by i__ a  __boy,__is a  beautiful girl.

2.BERT frame ：

pre-training and fine-tuning Two steps , First step , Training models with unlabeled data on different pre training tasks ; The second step , Use pre training initialization parameters and fine tune based on tagged data .

Task a ：Mask-L-M 15% Probability mask , among 80%mask,10% Random substitution ,10% unchanged

Task 2 ： Adjacent sentence prediction （next sentence prediction）,50% Adjacent sentences ,50% Random .

The first etymology of each sequence is special for classification Token[CLS], The state of the last hidden layer is related to this Token[CLS] Related to ,Token[CLS] It is used to generate the representation of sequences and for classification tasks , That is, the learned features are integrated into [CLS] In this etymology .

The training result of a large number of unlabeled data is not necessarily worse than that of a small number of labeled data sets .

Input consists of three parts ,token embedding Code words , Semantic information .segment Code the segment , It means from the first few sentences .position embedding Embed... For location , Represents location information .

3. Loss function

• The first part is from Mask-LM Word level classification task ; Using cross entropy loss
• The other part is the sentence level classification task ; Using cross entropy loss
• The total objective function is to minimize the sum of the two loss functions

 4. summary

Through the joint learning of these two tasks , You can make BERT The learned representations are token Level information , It also contains sentence level semantic information .

BERT Tell us , The training results on a large number of unlabeled data are not necessarily worse than those on a small number of labeled data .

BERT Use self-monitoring in two tasks （MLM and NSP） Pre training on , Make the encoder have high-quality feature extraction ability , The task of migrating to downstream is further completed NLP The task of .

BERT Intensive reading of the thesis paragraph by paragraph 【 Intensive reading 】_ Bili, Bili _bilibilihttps://www.bilibili.com/video/BV1PL411M7eQ?spm_id_from=333.999.0.0