【AI4Code】《Unified Pre-training for Program Understanding and Generation》 NAACL 2021

《Unified Pre-training for Program Understanding and Generation》 NAACL 2021

The article puts forward PLBART It is a sequence to sequence model , Able to perform a wide range of program and language understanding and generation tasks . PLBART adopt denoising autoencoding For a large number of Java and Python Functions and related NL The text is pre trained . Code summary 、 The experiments of code generation and code translation of seven programming languages show ,PLBART Superior to or comparable to the most advanced models . Besides , Experiments on discrimination tasks , For example, program repair 、 Clone detection and vulnerable code detection , Proved PLBART Effectiveness in program understanding . Besides , The analysis shows that PLBART Learn program grammar 、 style （ for example , Identifier naming convention ）、 Logical process （ for example ,else In block if Block is equivalent to else if block ） It is very important for program semantics , So it performs well even with limited comments .

Denoising pre training

PLBART be based on BARTbase framework , Use seq2seq Denoising pre training to take advantage of PL and NL Unlabeled data in , There are three noise strategies : Mark shielding 、 Mark deletion and mark filling , Input the noisy sequence encoder, Original sequence plus position offset input decoder, The goal is to remove noise and restore the original sequence .

among token infilling Yes, it will 0～k individual token Replace with [MASK] ,k=0 When you add a mask nothing more .

During pre training NL and PL The ratio is 1:14, Therefore, up sampling and down sampling are needed to remove bias .

Downstream tasks

Downstream tasks generate descriptions based on code , Generate code according to the description And code translation , It's all about seq2seq Mission .

There are also two classification tasks ： Clone code detection and fragile code detection , about pair Input , Join the two pieces of data , Use one in the middle </s> token Connect .decoder The last output of is sent to the linear classifier for classification .