当前位置:网站首页>NLP generation model 2017: Why are those in transformer
NLP generation model 2017: Why are those in transformer
2022-07-06 00:23:00 【Ninja luantaro】
1、 Briefly describe Transformer Feedforward neural network in ? What activation function is used ? Related advantages and disadvantages ?
Feedforward neural network adopts two linear transformations , The activation function is Relu, The formula is as follows :
F F N ( x ) = m a x ( 0 , x W 1 + b 1 ) W 2 + b 2 FFN(x) = max(0, xW_1 + b_1) W_2 + b_2 FFN(x)=max(0,xW1+b1)W2+b2
advantage :
- SGD The convergence rate ratio of the algorithm sigmoid and tanh fast ;( The gradient is not saturated , Solved the problem of gradient disappearance )
- Low computational complexity , There's no need to do exponential operations ;
- Suitable for backward propagation .
shortcoming :
- ReLU The output of is not zero-centered;
- ReLU It's very... During training ” fragile ”, Carelessness can cause neurons ” Necrosis ”. for instance : because ReLU stay x<0 Time gradient is 0, This leads to a negative gradient at this ReLU Set to zero , And this neuron may never be activated by any data . If this happens , So the gradient behind this neuron is always 0 了 , That is to say ReLU Neurons are dead , No longer responding to any data . In practice , If your learning rate It's big , It's very possible that you are on the Internet 40% All the neurons are dead . Of course , If you set a suitable smaller learning rate, This problem doesn't happen too often .,Dead ReLU Problem( Neuronal necrosis ): Some neurons may never be activated , Causes the corresponding parameters to never be updated ( In the negative part , The gradient of 0). There are two reasons for this phenomenon : Parameter initialization problem ;learning rate Too high results in too large parameter update during training . resolvent : use Xavier Initialization method , And avoid learning rate Set too large or use adagrad And so on learning rate The algorithm of .
- ReLU No amplitude compression of data , Therefore, the range of data will continue to expand with the increase of the number of model layers .
2、 Why Join in Layer normalization modular ?
motivation : because transformer Stacked A lot of layers , Easy to Gradient disappearance or gradient explosion ;
reason :
After the data passes through the function of the network layer , No longer normalized , The deviation will be bigger and bigger , So we need to data again Do normalization ;
Purpose :
Before the data is sent into the activation function normalization( normalization ) Before , You need to use the input information normalization Converted to an average of 0 The variance of 1 The data of , Avoid the input data falling in the saturation area of the activation function The gradient disappears problem
3、 Why? transformer Block use LayerNorm instead of BatchNorm?LayerNorm stay Transformer Where is the location of ?
Normalization The goal is to stabilize the distribution ( Reduce the variance of each dimension data ).
Another way to ask this question is : Why is image processing used batch normalization The effect is good , Natural language processing uses layer normalization good ?
LayerNorm Is to normalize the hidden layer state dimension , and batch norm Yes sample batch size Dimensions are normalized . stay NLP The task is not like the image task batch size Is constant , Usually changing , therefore batch norm The variance of will be larger . and layer norm Can alleviate the problem .
Reference material :
About Transformer Those why
边栏推荐
- FFmpeg学习——核心模块
- notepad++正則錶達式替換字符串
- MySql——CRUD
- [online chat] the original wechat applet can also reply to Facebook homepage messages!
- Detailed explanation of APP functions of door-to-door appointment service
- Reading notes of the beauty of programming
- Classical concurrency problem: the dining problem of philosophers
- 免费的聊天机器人API
- 《编程之美》读书笔记
- What is information security? What is included? What is the difference with network security?
猜你喜欢
Ffmpeg learning - core module
Configuring OSPF GR features for Huawei devices
Classical concurrency problem: the dining problem of philosophers
Browser local storage
Recognize the small experiment of extracting and displaying Mel spectrum (observe the difference between different y_axis and x_axis)
Spark SQL空值Null,NaN判断和处理
Go learning --- structure to map[string]interface{}
Start from the bottom structure and learn the introduction of fpga---fifo IP core and its key parameters
剖面测量之提取剖面数据
Hudi of data Lake (1): introduction to Hudi
随机推荐
FPGA内部硬件结构与代码的关系
Room cannot create an SQLite connection to verify the queries
How to use the flutter framework to develop and run small programs
Pointer pointer array, array pointer
Spark-SQL UDF函数
关于slmgr命令的那些事
选择致敬持续奋斗背后的精神——对话威尔价值观【第四期】
Tools to improve work efficiency: the idea of SQL batch generation tools
【DesignMode】组合模式(composite mode)
剖面测量之提取剖面数据
notepad++正则表达式替换字符串
wx.getLocation(Object object)申请方法,最新版
Key structure of ffmpeg -- AVCodecContext
Power query data format conversion, Split Merge extraction, delete duplicates, delete errors, transpose and reverse, perspective and reverse perspective
Global and Chinese market of valve institutions 2022-2028: Research Report on technology, participants, trends, market size and share
Calculate sha256 value of data or file based on crypto++
[Chongqing Guangdong education] reference materials for Zhengzhou Vocational College of finance, taxation and finance to play around the E-era
2022-02-13 work record -- PHP parsing rich text
行列式学习笔记(一)
Go learning --- structure to map[string]interface{}