Esmfold: a new breakthrough in protein structure prediction after alphafold2

just ,Meta AI Head of protein team Alexander Rives Announce the latest achievements on twitter ESMFold, High precision end-to-end atom level structure prediction can be carried out from single sequence of protein , Reasoning speed exceeds AlphaFold. This is by far the largest protein language model .

A year ago. ,DeepMind Open source AlphaFold2 Liandeng Nature、Science, Brush and explode creatures and AI educational circles . A year later ,Meta With an order of magnitude faster ESMFold coming . Not only fast , There are enough models 150 One hundred million parameters .

LeCun Tweet praise , This is a Meta-FAIR Great new achievements of protein team .

Work together Zeming Lin According to ,30 The big model with 100 million parameters is 256 individual GPU On training 3 A few weeks , and ESMfold stay 128 individual GPU Used it 10 God . as for 150 Billion parameter version , It's not clear yet .

He also said he , The code will definitely open source later , Stay tuned ！

Big and fast ！

today , Our main character is ESMFold, A sequence from a protein individual , Direct high accuracy 、 end-to-end 、 A model for predicting atomic hierarchy .

Address of thesis ：https://www.biorxiv.org/content/10.1101/2022.07.20.500902v1

150 The benefits brought by 100 million parameters are needless to say —— Through training , Today's large models can predict the three-dimensional structure of proteins with atomic size accuracy .

In terms of accuracy ,ESMFold and AlphaFold2、RoseTTAFold almost .

however ,ESMFold Guess it's faster than AlphaFold2 One order of magnitude fast ！

Let's talk about the order of magnitude. It may be difficult to understand the speed comparison between the three , Just look at the picture below .

What's the difference ？

although AlphaFold2 and RoseTTAFold We have made a breakthrough in the prediction of atomic resolution structure , But they also rely on the use of multiple sequence alignments （MSA） And similar protein structure templates to achieve the best performance .

by comparison , By using the internal representation of the language model ,ESMFold Using only one sequence as input, the corresponding structure prediction can be generated , Thus, the speed of structure prediction is greatly accelerated .

The researchers found that ,ESMFold The prediction of low complexity sequences is equivalent to the most advanced models .

and , The accuracy of structure prediction is closely related to the complexity of language model , in other words , When language models can better understand sequences , You can better understand the structure .

at present , There are billions of protein sequences whose structures and functions are unknown , Many of them come from metagenome sequencing .

utilize ESMFold, Researchers need only 6 Hours , It can be folded 100 Random samples of 10000 metagenomic sequences .

A large part of them have high confidence , And unlike any known structure （ There is no record in the database ）.

The researchers believe that ,ESMFold It can help understand the protein structure beyond cognition .

Besides , because ESMFold The prediction speed of is one order of magnitude faster than the existing models , So researchers can use ESMFold To help bridge the gap between the fast-growing protein sequence database and the slow-moving protein structure and function database .

150 Billion parameter protein language model

Next, let's talk about it in detail Meta This brand new ESMFold.

ESM-2 It's based on Transformer The language model of , And use the attention mechanism to learn the interaction mode between paired amino acids in the input sequence .

Compared with the previous generation model ESM-1b,Meta On the model structure 、 Training parameters have been improved , And add computing resources and data . meanwhile , Relative position embedded addition , So that the model can be extended to sequences of arbitrary length .

From the results , have 1.5 One hundred million parameters ESM-2 The model has 6.5 One hundred million parameters ESM-1b The model behaves better .

Besides , On the basis of structural prediction ,ESM-2 It also surpasses other protein language models . This performance improvement is consistent with the laws established in the field of large-scale language modeling .

With ESM-2 Increase in scale , It can be observed that the accuracy of language modeling has been greatly improved .

End to end single sequence structure prediction

SMFold and AlphaFold2 A key difference is ,ESMFold Use language model to express , Eliminated pairs of well-defined homologous sequences （ With MSA In the form of ） Need as input .

ESMFold By using a process sequence Transformer Module replacement processing MSA Computing expensive network modules , To simplify the AlphaFold2 Medium Evoformer. This simplification means ESMFold Greatly improve the speed of , Much higher than based on MSA Model of .

The output of the folding trunk is then processed by a structure module , It is responsible for outputting the final atomic level structure and the confidence of prediction .

Researchers ESMFold And AlphaFold2 and RoseTTAFold stay CAMEO（2022 year 4 Month to 2022 year 6 month ） and CASP14（2020 year 5 month ） Compare on the test set .

When only a single sequence is input ,ESMFold It's better than Alphafold 2 Much better .

When using a complete pipe ,AlphaFold2 stay CAMEO and CASP14 We have reached 88.3 and 84.7.ESMFold stay CAMEO Has made progress with RoseTTAfold Quite accurate , Its average TM The score is 82.0.

Conclusion

The researchers found that , Language models aimed at unsupervised learning are trained in a large, evolutionarily diverse protein sequence database , It can predict protein structure with atomic resolution .

Expand the parameters of the language model to 15B, We can systematically study the influence of scale on protein structure learning .

We see , The nonlinear curve of protein structure prediction is a function of model scale , We also observed a strong relationship between the understanding of sequences in language models and structural prediction .

ESM-2 The series of models is the largest protein language model trained so far , Its parameters are only one order of magnitude less than the recently developed largest text model .

and ,ESM-2 It is much better than the previous model , Even in 150M Under parameters ,ESM-2 Is better than ESM-1 Generation language model in 6.5 Capture a more accurate structural diagram under 100 million parameters .

The researchers say ,ESMFold The biggest driver of performance is the language model . Because there is a strong connection between the confusion of language model and the accuracy of structure prediction , They found that when ESM-2 When you can better understand the protein sequence , The prediction results equivalent to the current most advanced models can be obtained .

ESMFold Accurate atomic resolution structure prediction is obtained , Reasoning time is longer than AlphaFold2 It's an order of magnitude faster .

In practice , The advantage of speed is even greater . because ESMFold There is no need to search for sequences related to evolution to build MSA.

Although there are faster ways to reduce search time , But how to reduce it may still be a long time .

The benefits brought by the great shortening of reasoning time are self-evident —— The improvement of speed will make it possible to draw the structure space of large metagenomics sequence database .

In addition to structure based tools to identify remote homology and protection , use ESMFold Make fast and accurate structural prediction , It can also play an important role in the structural and functional analysis of a large number of new sequence sets .

Get millions of prediction structures in a limited time , It is conducive to the discovery of a new understanding of the breadth and diversity of natural proteins , And can discover new protein structure and protein function .

reference ：

Language models of protein sequences at the scale of evolution enable accurate structure prediction | bioRxiv