Transformer T5 model read slowly

Transformer T5 Read the model slowly

Code address ：https://github.com/google-research/text-to-text-transfer-transformer

Other reference addresses ：

Start with his thesis , But many pages ...

Address of thesis ：https://arxiv.org/abs/1910.10683 

One 、 A rough understanding of the model

T5 The origin of , Namely prompt Ideological evolution , Convert all downstream tasks into one task Let all tasks share one solution （ It greatly reduces the amount of parameters , If 2 A mission , Each task 10 Ten thousand parameters , Then this generative method is still 10 ten thousand , The traditional downstream model needs 20 Ten thousand parameters ）, Generative to generate answers . As for the loss function, it depends on the later design .（ What is shown in the following figure means the same ） 

Two 、 Some basic settings of the model

The dataset is C4 A big data set of web crawlers , The overall model is based on Transformer The generation model of .

1. Model framework ：

It is similar to classic Transformer The differences between models are 3 spot ,

① remove  Layer Norm bias, ② take layer normalization Put it outside the residual path , ③ Use different position embedding Strategy

Target loss form ： maximum likelihood objective (using “teacher forcing”）

2. Input form ：

① You can fine tune the downstream respectively , You don't need all downstream tasks to fine tune together .② utilize prefix Prefix hints to indicate different learning tasks .