当前位置：网站首页>RNN recurrent neural network

RNN recurrent neural network

2022-07-02 05:24:00 【TT ya】

Beginner little rookie , I hope it's like taking notes and recording what I've learned , Also hope to help the same entry-level people , I hope the big guys can help correct it ~ Tort made delete .

One 、RNN Significance of emergence

What we are familiar with CNN, Its output only considers the influence of the previous input and does not consider the influence of the input at other times （ That is, one input after another can only be processed alone ）

however , For some time related , A sequence of information （ That is, there is a relationship between before and after input ）, For example, predict the content of the document before and after , Now CNN The effect of is not very good .

Our cognition is based on past experience and memory , From this point of view and to the above CNN Inadequate remedy , The design not only considers the input of the previous moment , A recurrent neural network that can also remember the contents in front of the network ——RNN.

Two 、RNN principle

1、RNN Model structure and forward propagation

RNN By input layer , Composition of hidden layer and output layer .

among x,s,o Is a vector , They are the values of the input layer , Hidden layer value and output layer value .

U Is the weight matrix from the input layer to the hidden layer ,V Is the weight matrix from hidden layer to output layer ,W Is the weight matrix of the last value of the hidden layer as the input of this time .

The formula is as follows ：

$\begin{array}{l} O_{t}=g\left(V \cdot S_{t}\right) \\ S_{t}=f\left(U \cdot X_{t}+W \cdot S_{t-1}\right) \end{array}$

among f and g Is the activation function ,f It can be tanh,relu,sigmoid Wait for the activation function , and g Usually softmax.

ad locum U,V,W It is the same. （ Back propagation and then change , This is just to emphasize that the variable is the latter 3 individual ）, The change is Xt,St-1 and St, there W*St-1 It is the influence of the value of the previous moment （ The so-called memory of the past ） Join in .

Specifically, as shown in the figure below, expand by time

2、 Back propagation

Output value of each time Ot Will produce an error value Et

The loss function can use either cross entropy loss function or square error loss function

First let's look at the formula ：

Total error

$E=\sum_{t} e_{t}$

Parameter gradient method

$\nabla U=\frac{\partial E}{\partial U}=\sum_{t} \frac{\partial e_{t}}{\partial U}$

$\nabla V=\frac{\partial E}{\partial V}=\sum_{t} \frac{\partial e_{t}}{\partial V}$

$\nabla W=\frac{\partial E}{\partial W}=\sum_{t} \frac{\partial e_{t}}{\partial W}$

From the above formula, we can get its meaning ： The sum of the partial derivatives of the deviation at each moment （U,V,W It's all like this ）

We'll take W For reference ：

First, expand the formula with the chain rule

$\frac{\partial E_{t}}{\partial W}=\frac{\partial E_{t}}{\partial o_{t}} \frac{\partial o_{t}}{\partial s_{t}} \frac{\partial s_{t}}{\partial W}$

Then from the formula just $s_{t}=f\left(U x_{t}+W s_{t-1}\right)$ Plug in , And found st With all the previous moments s There is a direct or indirect relationship , We can get the following formula ：

$\frac{\partial E_{t}}{\partial W}=\sum_{k=0}^{t} \frac{\partial E_{t}}{\partial o_{t}} \frac{\partial o_{t}}{\partial s_{t}} \frac{\partial s_{t}}{\partial s_{k}} \frac{\partial s_{k}^{+}}{\partial W}$

V and U The formula is as follows

$\frac{\partial E_{t}}{\partial V}=\frac{\partial E_{t}}{\partial O_{t}} * \frac{\partial O_{t}}{\partial V}$

$\frac{\partial E_{t}}{\partial U}=\sum_{k=0}^{t} \frac{\partial E_{t}}{\partial o_{t}} \frac{\partial o_{t}}{\partial s_{t}} \frac{\partial\left(W^{t-k} a_{k}\right)}{\partial U}$

3、 ... and 、RNN Application and deficiency of

1、RNN Application field

natural language processing (NLP): Mainly video processing , The text generated , Language model , The image processing

Machine translation , Text similarity calculation , Image description generation

speech recognition

recommend

2、 Insufficient

It is easy to have the problem of gradient disappearance or gradient explosion .

reason ： Long time dependence leads to over fitting, which leads to gradient explosion, and long time leads to small memory value, which leads to gradient disappearance .

You are welcome to criticize and correct in the comment area , thank you ~

原网站

版权声明
本文为[TT ya]所创，转载请带上原文链接，感谢
https://yzsam.com/2022/02/202202151841030659.html