NLP commonly used Backbone model cheat sheet (1)

Foreword

Since the appearance of Transformer in 2017, it has appeared in all major NLP jobs.Recently, Stanford also opened a course CS25 specifically for transformers: [Stanford] CS25 Transformers United | Fall 2021

People who are new to NLP can read an article I wrote earlier Research 0_NLPer set off

For the corresponding model, you can go to hugginface's transformers library to see transformers/models (github), you can find the corresponding model to see its source code implementation.

Now it is mainly the dynamic word vector coding technology combined with the context, and the word2vec and glove vocabulary are rarely used for static word vector mapping.

B station a video Blow up!Doctor of Computer [NLP Natural Language Processing] is worthy of being a professor of Tsinghua University!5 hours got me done with NLP Natural Language Processing! (虽然标题有些emm…但是看了一下目录啥的好像还行…

Thesis

Mass

Bart

T5

Exploring the Limits of Transfer Learning with a Unified
Text-to-Text Transformer

Bert

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

encoder structure.There are many bert families, such as the distilled version distilBert, the variant Roberta, etc.

Word vector input composition:

transformer

The famous self-attention comes from this article.
Attention Is All You Need

This model has been reproduced before: Transformer structure reproduction__attention is all you need (pytorch)

encoder-decoder structure:

Attention模块: