【AI4Code】《IntelliCode Compose: Code Generation using Transformer》 ESEC/FSE 2020

《IntelliCode Compose: Code Generation using Transformer》 ESEC/FSE 2020

It's not just about generating a word , Instead, generate a line . It's using GPT-2. The dataset is 12 Billion rows Python, C#, Javascript, TypeScript Code of language

Byte-Pair Encoding (BPE)

On the sequence token Chemical treatment , One is to use subtoken To narrow the vocabulary , One is to shield strings to prevent sensitive data leakage .

IntelliCode Compose

The model uses GPT, When inferring, it will sequence decoding The process of is regarded as the search process of the tree , until token appear ：

Use when building trees beam search,beam with by K, Suppose the length of the final generated sequence is L, The model needs to predict K*L Time , But the model can batch perform , So all you need is L Time .

Multilingual model

Four ways of modeling multilingualism are compared ：

1） Ignore differences between languages , Train multiple languages with a unified model 【 Experiments show that ： This method is worse than monolingual training alone 】

2） Join in language type embedding Information , Each language is represented by a vector , And the original token embedding Equal combination .

3） Add a sentence at the beginning of each training sample "lang * remaining token sequence", among $lang\in \{Python,C\#,JavaScript,TypeScript\}$

4） In pre training , Add a language type classification Mission , That is, one more head, Predict the type of the language each time .