Bert's summary of me

BERT Read a lot of information , I think I have some insight . For two years , I didn't sort it out myself . Now start sorting ：

One 、Google Bert In the source modeling file
modeling yes bert The origin of , It's better to understand here first . You can refer to the materials of other great gods ：
1. Code interpretation , Analysis of a three-year-old brother , It's very clear
2. bert The paper of , The first article should read 《BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding》, This link has a good explanation in Chinese Add link description
3. The second important paper is 《Pre-Training with Whole Word Maskingfor Chinese BERT》. The idea is google Bring up the , The Chinese version was trained by Harbin Institute of technology , Hada's link is This github. Relevant supporting materials include ：BERT-WWM note 、BERT-wwm、BERT-wwm-ext
4. Met a summary BERT Information articles , Look at this link . however , I think he wrote too much , This means that these things are not necessarily useful .
Two 、transform You should make a good understanding of
2.1 The first one is Wang Yudi's pdf, It's really good . After seeing , combination tensorflow Code , View paper Attention Is All You Need
3、 ... and 、 How to load the code in the application ？
（1）keras The loading method is simple , There is a tool developed by sujianlin's team . See here for its use ： Introduction 、github Address
（2）huggingface Of github see here ,Google Officially recommended PyTorch BERB Version implementation . For example , see B The graduate student at the station Example , You can also learn by hand Bert Text classification of this Example
（3） Official Google Code , It seems that loading is also good , Sure

Four 、 Other matters needing attention
（1） Optimizer used adamw, It is different from the conventional adam What improvements have been made , see here