[5 minutes paper] Pointer network

The problem solved ？

Propose a network structure , Learn the positional relationship of input sequence .

background

Learning the positional relationship of input sequences can be regarded as seq2seq problem , The length of the output sequence is consistent with the length of the input sequence , And it's a variable . It can be used to deal with variable sequencing or combinatorial optimization problems .

This article is 2017 Published in , before this , For algorithms such as recurrent neural networks , Mainly based on a priori fixed output dictionary , That is, the predicted size of the output unit of the hidden layer is unknown .

The method adopted ？

Compared with the previous attention Used to code encoder The hidden state of the input is different , Proposed by the author Ptr-Net Is based on Attention To choose .

The core network structure in this paper is decoder part . There are two main points ：

1. In the past seq2seq Different models ,decoder The input of each time is actually the input information of the node selected at the previous time , It is the most original coordinate information encoded and then enters encoder Input information in .
2. In the calculation decoder When the output of , Adopted attention The calculation of is as follows ： Input is [bs, hidden_dim, seq_len] Of context And the dimension is [bs, hidden_dim] Of input. there input Namely decoder The input and hidden layer of LSTM The resulting output .

context After a one-dimensional convolution operation , It is also equivalent to passing through a weight matrix $W_1$ obtain [bs, hidden_dim, seq_len] Dimensionally invariant matrix .input Through a weight matrix $W_2$ Extend dimension to [bs, hidden_dim, seq_len]. Then these two matrices are added together with one [bs, 1, hidden_dim] Learnable variables of $V$ Multiply . obtain [bs, seq] Dimensional attention matrix . Corresponding to the original formula is :

$$u_j^i=v^{T}tanh(W_1{e}_j+W_2{d}_i)j\in (1,\cdots n)$$

Go over one floor softmax obtain attention matrix ：

$$a_j^i=softmax(u_j^i)j\in (1,\cdots n)$$

This [bs, seq] Dimensional attention The matrix and dimension are [bs, hidden_dim, seq_len] Of context Multiply the matrix to get the output of the hidden layer [bs, hidden_dim], As LSTM The next moment of hidden state.

$$d_i^{\prime }=\sum _{j=1}^n{a}_j^i{e}_j$$

The result is ？

Published information ？ The author information ？

Reference link