Deep learning: gat

Original thesis
Explain in detail ：E:\ The group will report \2021.11.5 report -GAT\20211105 draft
May refer to ： Detailed introduction gac And improvement process

source ： about GCN Improvement

Attention mechanism

The attention mechanism in deep learning is essentially similar to the selective visual attention mechanism of human beings , The core goal is to select the information that is more critical to the current task goal from a large number of information .

GCN limited

• It is difficult to assign different weights to different neighbor.
  This limits the ability of the model to capture the relevance of spatial information , This is also inferior to GAT Root cause of .

stay GAT in , Each node in the graph can be based on the characteristics of adjacent nodes , Assign different weights to it

• GCN The way of combining the characteristics of adjacent nodes is related to the structure of the graph
  This limits the generalization ability of the trained model on other graph structures .

GCN The limitations are : Hard to handle dynamic graphs ; It is difficult to assign different weights to different neighbor. Actually GAT yes GNN Improvement , And GCN similar , It's just based on self-attention The graph model of .

difference

The core difference lies in how to collect and accumulate distance 1 Characteristic representation of neighbor nodes of .

Figure attention model GAT Replace with attention mechanism GCN Fixed standardized operation in .
Essentially ,GAT Just put the original GCN The standardized function of is replaced by the neighbor node feature aggregation function using attention weight .

Purpose

GAT And GCN It is also a feature extractor , Aiming at N Nodes , Predict and output the characteristics of new nodes according to the input node characteristics .

GAT characteristic

• 1 Each node in the graph can be based on the characteristics of adjacent nodes , Assign different weights to it
• 2 Only related to adjacent nodes , That is, the nodes of the shared edge , There is no need to get the information of the whole picture ：
  (1) The graph does not need to be undirected ( If the edge j → i non-existent , We can simply omit the calculation α i j;
  (2) It makes our technology directly applicable to inductive learning—— Include the task of evaluating models on graphics that are completely invisible during training .

The basic idea

According to the position of each node on its adjacent nodes attention, To update the node representation .

The process

this paper : Assign corresponding weights to different adjacent nodes , multi-head Bulls Attention structure , Calculate the attention coefficient

For vertices i , Calculate its neighbors and... One by one i The similarity coefficient between :

That is, first use the shared parameters W Add dimension to vertices , Post stitching (concatenate) Two characteristics , By mapping functions g(.) Map high-dimensional features to a Attention In real numbers .

Through to i Our neighbors softmax, You can get ( Learning out ) The relationship coefficient between nodes :

To sum by weight aggregate

Generally speaking , The aggregation method generally weights and sums the features transmitted by neighbors , You can update the characteristics of this node :

In this article, the way of collection is enhanced , use K An attention mechanism , That's it K A neighbor weighting method , To update the characteristics of this node . namely Attention Medium multi-head thought :

Model comparison

Case study ： The paper cora

GCN-dgl

Code location ：E:\ Project routines \GCN\ The paper \ Code location

Data presentation

result

Official website ：
Code ：E:\ Project routines \GCN\ The paper \gcn_dgl_ Official website

GAT-dgl

Code ：

result