KDD 2022 | graph neural network generalization framework under the paradigm of "pre training, prompting and fine tuning"

This article is about SIGKDD 2022 Selected papers “GPPT: Graph Pre-training and Prompt Tuning to Generalize Graph Neural Networks” The interpretation of . This paper was completed by the research group of Professor Wang Ying, School of computer science and technology, Jilin University .

This paper proposes for the first time “Pre-training、Prompt、Fine-tuning” The concept of refactoring downstream tasks , Make it have a connection with Pretext Similar mission objectives , make up GNN The task gap , Solution by tradition GNN In pre training Pretext It is difficult to elicit pre trained graph knowledge due to the inherent training target gap between tasks and downstream tasks 、 The problem of negative transfer . Experiments show that , This training strategy is superior to all other training strategies , Including supervised learning 、 Joint training and traditional transfer learning .

 

Paper title ：

GPPT: Graph Pre-training and Prompt Tuning to Generalize Graph Neural Networks

Research background

Figure neural network (GNNs) It has become a technique for analyzing graph structure data in many real-world systems , Including social networks 、 Recommend systems and knowledge mapping .GNN The general approach of treats the input as an underlying computational graph , Learn node representations by passing messages across edges . The generated node representation can be used for different downstream tasks , Such as link prediction 、 Node classification and node attribute fitting .

Recently, in the field of transfer learning, we have made GNN Capture transportable graph patterns to extend to different downstream tasks . say concretely , Most follow “ To train in advance 、 fine-tuning ” Learning strategies ： Use easily accessible information as Pretext Mission （ Such as edge prediction ） Yes GNN pretraining , Use the pre trained model as initialization to fine tune the downstream tasks .

Problems and challenges

The paper pays attention to the tradition GNN In pre training Pretext Internal training goal gap between tasks and downstream tasks , Not only may it not be possible to derive the pre trained graph knowledge , It may even lead to negative migration . Besides ,Pretext The task requires both expertise , It also requires tedious manual tests . therefore , For the first time, the paper puts forward “Pre-training、Prompt、Fine-tuning” The concept of refactoring downstream tasks , Make it with Pretext Target tasks with similar tasks , To bridge the task gap between the pre training goal and the fine-tuning goal .

To overcome tradition “Pre-training、Fine-tuning” The limitations of , It draws lessons from natural language processing “Prompt” technology . Because the prompt tuning is NLP Technology unique to the field , Therefore, it is difficult to design suitable for GNN Of Prompt Templates . The paper overcomes two main challenges ：1） How to apply semantic prompt function to reconstruct various graph machine learning tasks in graph data ;2） How to design Prompt Templates to better redesign downstream applications , Propose graph pre training and prompt tuning (GPPT) frame .

Method

First , use Masked Edge Prediction The task is right GNN pretraining , The downstream node classification task is reconstructed into a link prediction task . then , In order to narrow the gap between the pre training goal and the downstream task goal , Using paired token templates Graph Prompt Function to change an independent node to a marked pair , Each tag pair contains a task token representing the downstream problem （task token） And a structure token containing node information （structure token）.

Task token （ Indicates the node label ） And structure token （ Description node ） It can be directly used to fine tune the pre training model without changing the classification layer . then , The node classification method is reformulated by using the node link prediction score , The task tag with the highest score is determined as the node tag . Last , Through experiments, the proposed GPPT Supervised learning 、 The effectiveness of joint training and traditional transfer learning , And the advantages of this learning mode under the small sample setting .

experiment

We are 8 The proposed framework is evaluated on a popular benchmark dataset GPPT, Including citation network (Cora、Citeseer、Pubmed)、Reddit、CoraFull、Amazon-CoBuy（Computer and Photo）、Ogbn-arxiv.

Cue based learning methods usually get the best performance on benchmarks , Using graph clustering and neighborhood structure information is Prompt The key to token design .

summary

We creatively put forward GPPT, The first one is for GNN Conduct “ Preliminary training 、 Tips 、 fine-tuning ” The paradigm of transfer learning . The graph prompt function for graph data is designed for the first time , To reformulate and Pretext Downstream tasks with similar tasks , So as to reduce the gap between the two training goals . meanwhile , We also designed a task and structure token generation method , Used to generate node tips in the node classification task . Besides , We propose average hint initialization and orthogonal regularization methods to improve hint tuning performance . A lot of experiments show that ,GPPT It is superior to the traditional training paradigm on the benchmark data set , At the same time, it improves the tuning efficiency and better adaptability to downstream tasks . In the future work , We will explore the prompting function of the graph in the more challenging knowledge graph , And try to improve prompt tuning through meta learning .