[ai4code final chapter] alphacode: competition level code generation with alphacode (deepmind)

AlphaCode—— Macro recommendations （DeepMind）

14 Work together ,74 Page papers .

Think CodeX Just made a simple Natural language - Programming language My translation task ,AlphaCode To do a more difficult . The input and output are ：

Method

technological process

Model training is divided into pre training and fine tuning , Then, in the prediction stage, large-scale sampling （ Recall ） Get onemillion , Then clustering and filtering get 1000 individual （ Rough row ）, Then choose 10 Submission （ Fine discharge ）.

Data sets

First in Github Collect open source code , After pretreatment and cleaning 715GB, As a pre training data set ; And then use CodeContests Dataset tuning , The format is shown in the figure above .

Model structure

No model diagram . differ CodeX Of GPT, Only Transformer Of decoder, Here is a complete Transformer, both encoder Also have decoder. Minimum model 3 Million parameters , Maximum model 4000 Million parameters .

What is worth mentioning here is the multi-head attention Only multiple query,KV It's all the same .

Fine-tuning

encoder The input is the description of the topic （ Also include ： Of the subject tags、solution The language used 、 Examples in the title ）,decoder The output of corresponds to ground-truth It's a solution, It can be right , It can also be wrong .

Sampling & Evaluation： Massive trial and error

• Step1: Enter the title description into Model 1, From the model 1 Sample out 100 Ten thousand output codes .
• Step2: Obviously, the sampled 100 In 10000 codes 99% Can't run 、 Wrong code , Use the one that comes with the title test case, First filter out these invalid 、 Error code , Still left 1000 Code that can run .（ Recall ： One million ～ thousand ）
• Step3: AlphaCode Introduce an extra Model 2, Model 2 It's also a model 1 Pre training model of , however fine-tuning The purpose of is to enter the Title Description , Output test case. Model 2 Automatically generated test case Accuracy is not guaranteed , It's just For next clustering use Of . Model 2 Generated for the current problem 50 individual test inputs.
• Step4: The generated 50 individual test inputs Input to 1000 Code , If the generation results of some codes are almost the same , Explain that the algorithm or logic behind these codes is similar , Can be classified into one category . After clustering , leave 10 Class code , First select the code from a large number of classes to submit , More likely to win .（ Fine discharge ： thousand ～10）

experimental result

Competition results

stay 10 The result of this program competition . Did not actually participate , It's the estimated ranking , Because there are penalties in the competition , So this penalty can only be estimated .AlphaCode Basically in Codeforce Medium level in the competition .

Evaluation indicators

• [email protected]：Step1 Sampling recall K Code , Then cluster and select 10 Submission .
• [email protected]： Sample out k Code , If one is right, it will be a hit .

The effect of the number of samples

Step1 The more code samples are recalled , The better the result. . The comparison between the left and right figures shows , After filtering 、 Cluster the selected code , Basically the best . Especially the comparison of the following figure ：

Random from K It's useless to choose among the samples ; Clustering is a little better than filtering alone ; clustering + Filtering and oracle It's a little bit close .

The influence of money

The longer you train , The more you pick , The better the result. .

Security discussion

There's only one page （ article 74 page ）, Basic is CodeX Subset .