Gru of RNN

LSTM Have longer memory ability , In most sequential tasks, we have achieved better results than the basic RNN Better performance of the model ,LSTM Gradient dispersion is not easy to occur . however LSTM The structure is relatively complex , It's expensive to calculate , The amount of model parameters is large .

One 、GRU principle

  The study found that , The forgetting door is LSTM The most important gating in , It is even found that the network with only forgetting gate outperforms the standard on multiple benchmark data sets LSTM The Internet . In simplified version LSTM in , Gated loop network (Gated Recurrent Unit, abbreviation GRU) Is the most widely used RNN One of the variants . GRU Merge the internal state vector and the output vector , Unified as state vector , door The control quantity is also reduced to 2 individual ： Reset door (Reset Gate) And update the door (Update Gate)

1. Reset door

The reset gate is used to control the status of the last timestamp Get into GRU The amount of .

Gating vector Entered by the current timestamp and Last timestamp status The result of change , Relationship ：. Gating vector Control status only , Without controlling the input ：. When = 0 when , New input All from input , Don't accept , This is equivalent to resetting . When = 1 when , And the input Together produce new output

 

2. Update door

Update the last timestamp status of door control And Xintu For the new state vector The degree of influence . Update gating vector from obtain .

  be used for Control new input The signal , be used for Control the last time status The signal ：

and Yes The number of updates is competing with each other 、 A state of ebb and flow . When updating the door = 0 when , All from the last timestamp status ; When updating the door = 1 when , All from new losses

Two 、GRU Realization

1.GRUCell

GRUCell and SimpleRNNCell、LSTMCell Use the same , establish GRU Cell object , And circularly expand the operation on the time axis . It needs to maintain only one initialization state vector ( monolayer )

x = tf.random.normal([2, 80, 100])
#  Initialize the state vector , GRU  only one 
h = [tf.zeros([2, 64])]
cell = layers.GRUCell(64)  #  newly build  GRU Cell, The length of the vector is  64
#  Untie in the timestamp dimension , Cycle through  cell
for xt in tf.unstack(x, axis=1):
    out, h = cell(xt, h)

 2.GRU

adopt layers.GRU Class can be easily passed through Sequential Containers can be stacked in multiple layers GRU Layer network , And do not maintain the initialization state vector

net = keras.Sequential([
layers.GRU(64, return_sequences=True),
layers.GRU(64)
])
out = net(x)