KDD 2022 | graph neural network generalization framework under the paradigm of "pre training, prompting and fine tuning"

author | Social media school SMP

source  | Social media school SMP

This article is about SIGKDD 2022 Selected papers “GPPT: Graph Pre-training and Prompt Tuning to Generalize Graph Neural Networks” The interpretation of . This paper was completed by the research group of Professor Wang Ying, School of computer science and technology, Jilin University .

This paper proposes for the first time “Pre-training、Prompt、Fine-tuning” The concept of refactoring downstream tasks , Make it have a connection with Pretext Similar mission objectives , make up GNN The task gap , Solution by tradition GNN In pre training Pretext It is difficult to elicit pre trained graph knowledge due to the inherent training target gap between tasks and downstream tasks 、 The problem of negative transfer . Experiments show that , This training strategy is superior to all other training strategies , Including supervised learning 、 Joint training and traditional transfer learning .

Paper title ：

GPPT: Graph Pre-training and Prompt Tuning to Generalize Graph Neural Networks

Research background

Figure neural network (GNNs) It has become a technique for analyzing graph structure data in many real-world systems , Including social networks 、 Recommend systems and knowledge mapping .GNN The general approach of treats the input as an underlying computational graph , Learn node representations by passing messages across edges . The generated node representation can be used for different downstream tasks , Such as link prediction 、 Node classification and node attribute fitting .

Recently, in the field of transfer learning, we have made GNN Capture transportable graph patterns to extend to different downstream tasks . say concretely , Most follow “ To train in advance 、 fine-tuning ” Learning strategies ： Use easily accessible information as Pretext Mission （ Such as edge prediction ） Yes GNN pretraining , Use the pre trained model as initialization to fine tune the downstream tasks .

Problems and challenges

The paper pays attention to the tradition GNN In pre training Pretext Internal training goal gap between tasks and downstream tasks , Not only may it not be possible to derive the pre trained graph knowledge , It may even lead to negative migration . Besides ,Pretext The task requires both expertise , It also requires tedious manual tests . therefore , For the first time, the paper puts forward “Pre-training、Prompt、Fine-tuning” The concept of refactoring downstream tasks , Make it with Pretext Target tasks with similar tasks , To bridge the task gap between the pre training goal and the fine-tuning goal .

To overcome tradition “Pre-training、Fine-tuning” The limitations of , It draws lessons from natural language processing “Prompt” technology . Because the prompt tuning is NLP Technology unique to the field , Therefore, it is difficult to design suitable for GNN Of Prompt Templates . The paper overcomes two main challenges ：1） How to apply semantic prompt function to reconstruct various graph machine learning tasks in graph data ;2） How to design Prompt Templates to better redesign downstream applications , Propose graph pre training and prompt tuning (GPPT) frame .

Method

First , use Masked Edge Prediction The task is right GNN pretraining , The downstream node classification task is reconstructed into a link prediction task . then , In order to narrow the gap between the pre training goal and the downstream task goal , Using paired token templates Graph Prompt Function to change an independent node to a marked pair , Each tag pair contains a task token representing the downstream problem （task token） And a structure token containing node information （structure token）.

Task token （ Indicates the node label ） And structure token （ Description node ） It can be directly used to fine tune the pre training model without changing the classification layer . then , The node classification method is reformulated by using the node link prediction score , The task tag with the highest score is determined as the node tag . Last , Through experiments, the proposed GPPT Supervised learning 、 The effectiveness of joint training and traditional transfer learning , And the advantages of this learning mode under the small sample setting .

experiment

We are 8 The proposed framework is evaluated on a popular benchmark dataset GPPT, Including citation network (Cora、Citeseer、Pubmed)、Reddit、CoraFull、Amazon-CoBuy（Computer and Photo）、Ogbn-arxiv.

Cue based learning methods usually get the best performance on benchmarks , Using graph clustering and neighborhood structure information is Prompt The key to token design .

summary

We creatively put forward GPPT, The first one is for GNN Conduct “ Preliminary training 、 Tips 、 fine-tuning ” The paradigm of transfer learning . The graph prompt function for graph data is designed for the first time , To reformulate and Pretext Downstream tasks with similar tasks , So as to reduce the gap between the two training goals . meanwhile , We also designed a task and structure token generation method , Used to generate node tips in the node classification task . Besides , We propose average hint initialization and orthogonal regularization methods to improve hint tuning performance . A lot of experiments show that ,GPPT It is superior to the traditional training paradigm on the benchmark data set , At the same time, it improves the tuning efficiency and better adaptability to downstream tasks . In the future work , We will explore the prompting function of the graph in the more challenging knowledge graph , And try to improve prompt tuning through meta learning .

Read more

# cast draft   through Avenue #

  Let your words be seen by more people  

How to make more high-quality content reach the reader group in a shorter path , How about reducing the cost of finding quality content for readers ？ The answer is ： People you don't know .

There are always people you don't know , Know what you want to know .PaperWeekly Maybe it could be a bridge , Push different backgrounds 、 Scholars and academic inspiration in different directions collide with each other , There are more possibilities . 

PaperWeekly Encourage university laboratories or individuals to , Share all kinds of quality content on our platform , It can be Interpretation of the latest paper , It can also be Analysis of academic hot spots 、 Scientific research experience or Competition experience explanation etc. . We have only one purpose , Let knowledge really flow .

  The basic requirements of the manuscript ：

• The article is really personal Original works , Not published in public channels , For example, articles published or to be published on other platforms , Please clearly mark  

• It is suggested that  markdown  Format writing , The pictures are sent as attachments , The picture should be clear , No copyright issues

• PaperWeekly Respect the right of authorship , And will be adopted for each original first manuscript , Provide Competitive remuneration in the industry , Specifically, according to the amount of reading and the quality of the article, the ladder system is used for settlement

  Contribution channel ：

• Send email ：[email protected] 

• Please note your immediate contact information （ WeChat ）, So that we can contact the author as soon as we choose the manuscript

• You can also directly add Xiaobian wechat （pwbot02） Quick contribution , remarks ： full name - contribute

△ Long press add PaperWeekly Small make up

Now? , stay 「 You know 」 We can also be found

Go to Zhihu home page and search 「PaperWeekly」

Click on 「 Focus on 」 Subscribe to our column

·

·