[pytorch] LSTM neural network

Processing time series data , There has been a RNN Cyclic neural network and GRU neural network Two classic Networks . Of course, there is another LSTM neural network , Long and short term memory neural network .
From the perspective of development history , It is available LSTM There's more GRU Of , But in terms of complexity ,LSTM Than GRU More complicated .

Let's recall GRU, It has two doors （ Update doors and reset doors ）, There is a vector that records historical information $H_t$.
and LSTM It's more complicated , Whether in the number of doors or the vector recording historical information .

LSTM neural network

There are 3 A door ,2 Status .

Control door

Oblivion gate

This with GRU The reset door in is very similar , The meaning is also roughly the same .
$$F_t=\Theta \left(X_t\cdot W_{xf}+H_{t-1}\cdot W_{hf}+b_f\right)$$
Then ,$F_t$ Act on memory $C_{t-1}$, Forget some historical information .

Input gate

It is associated with GRU The update door in is a little similar , But it's not exactly the same .
because LSTM There are two states , It can be understood as $C_t$ The renewal door of .
$$I_t=\Theta \left(X_t\cdot W_{xi}+H_{t-1}\cdot W_{hi}+b_i\right)$$
Then it will act on candidate memory $C_t^{\prime }$, Update to get new $C_t$.

Output gate

It is associated with GRU The update door in is a little similar , But it's not exactly the same .
because LSTM There are two states , It can be understood as $H_t$ The renewal door of .
$$O_t=\Theta \left(X_t\cdot W_{xo}+H_{t-1}\cdot W_{ho}+b_o\right)$$
Then it will act on candidate memory $C_t$, Update to get new $H_t$.

state

Memory state

You can see from the whole update process ,$C_{t-1}$ Forget some information first , And then with the candidate memory （ according to $X_t$） The generated part of the information is merged , obtain $C_t$.
Its change is relatively slow , It is also called long-term memory .

Hidden state

$H_t$ According to the current output （$X_t$ And $H_{t-1}$ Result ） And current memory $C_t$ The result of the action . Compared with $C_t$,$H_t$ And $H_{t-1}$ Weaker relationship , therefore $H_t$ Change faster . Therefore, it is also called short-term memory .

Combine the above two states ： Long term memory and short-term memory , It is called short-term and long-term memory neural network .

Code implementation

pytorch It also provides for LSTM layer , It is very convenient to call .
But you need to define the initial state value yourself （ A binary ）.

class LSMT_Net(nn.Module):
    def __init__(self, vocab_size, hidden_size, **kwargs):
        super(LSMT_Net, self).__init__(**kwargs)
        self.vocab_size = vocab_size
        self.hidden_size = hidden_size
        self.LSMTlayer = nn.LSTM(vocab_size , hidden_size, num_layers= 2)
        self.L1 = nn.Linear(hidden_size , vocab_size)
    def forward(self, inputs, state):
        X = F.one_hot(inputs.T.long(), self.vocab_size) #  Turn into one and only one 1, The rest are 0 Vector 
        X = X.to(torch.float32)        
        Y , state = self.LSMTlayer(X , state)
        Y = Y.reshape((-1 , Y.shape[-1]))
        Y = self.L1(Y)
        return Y , state
    def begin_state(self , batch_size):
        return (torch.zeros(self.LSMTlayer.num_layers , batch_size , self.hidden_size),
               torch.zeros(self.LSMTlayer.num_layers , batch_size , self.hidden_size))