Mlperf training v2.0 list released, with the same GPU configuration, the performance of Baidu PaddlePaddle ranks first in the world

This article has been published on the official account of the flying oar , Please check the link ：
MLPerf Training v2.0 The list is released , In the same way GPU The performance of Baidu PaddlePaddle under configuration is the first in the world

stay 6 month 30 The latest MLPerf Training v2.0 In the list , Baidu uses the propeller frame （PaddlePaddle） And Baidu AI Cloud Baige computing platform BERT Large Model GPU Training performance results , In the same way GPU Ranked first in all submission results under the configuration , It has surpassed highly customized optimization and has been in the leading position in the list for a long time NGC PyTorch frame , It shows the performance advantages of the propeller frame to the world .

chart 1 MLPerf Training v2.0 BERT Top five training results of model effectiveness

chart 1 It shows MLPerf Training v2.0 BERT Model in 8 card NVIDIA GPU A100（400W Power waste ,80G memory ） The training performance results of the top five , The Baidu PaddlePaddle plan is faster than other submitted results 5%-11% Unequal .

“ World first ” The black technology behind it

Where are the oars BERT Model 8 card GPU Training has created the world's best training performance , This comes from the basic performance of the propeller frame and the leadership of Distributed Technology , And the propeller and NVIDIA GPU Deep collaborative optimization .

For deep learning model training tasks , From data reading to model calculation , From the bottom operator to the upper distributed strategy , From multi device load balancing to whole process scheduling mechanism , Will affect the final training performance . The propeller is based on leading architectural design and long-term practice , Systematic optimization work has been made in high-performance training , Mainly reflected in the following aspects ：

Load balancing of data reading and model training

Aiming at the problem of load imbalance that often occurs in distributed training , Train the model and read the data 、 Pretreatment is allocated to different devices , Ensure that heterogeneous computing power makes the best use of everything , Implementation data IO And the balance of calculation .

Calculation acceleration of variable length sequence input model

For variable length sequence input models, most of them adopt padding The problem of redundant computation caused by filling alignment , Provide efficient support for variable length input and corresponding model structure , Give Way GPU Computing resources focus on efficient computing , Especially for Transformer The calculation efficiency of class model is significantly improved .

High performance operator library and fusion Optimization Technology

For the ultimate demand of framework foundation performance optimization , Developed a high-performance operator Library PHI, Fully optimize GPU Kernel Implementation , Improve the parallelism of the internal calculation of the operator , And through operator fusion to reduce the imitation memory overhead , Develop GPU The ultimate performance of .

High speedup hybrid parallel training strategy

For traditional data parallel performance 、 The bottleneck of video memory is limited , It realizes the parallel of fused data 、 Model parallel 、 A hybrid parallel distributed training strategy of grouping parameter slicing parallel strategy , Distributed training performance with Superlinear acceleration can be achieved in some scenarios .

Asynchronous scheduling of the whole process

The synchronization frequency of each link in the model training process is high 、 Low degree of time overlap , Design Asynchronous scheduling mechanism , Most of the synchronization operations are removed while ensuring the convergence of the model , Data processing 、 Training and collective communication are almost asynchronous scheduling , Improve end-to-end performance .

Help big model Technological innovation and industrial landing

Baidu has always attached importance to the technological research and development of large models , And is committed to promoting the industrial landing of large models . Large model training requires deep learning framework to provide strong support in high-performance distributed training .

Propeller distributed training starts from industrial practice , Continuously strengthen the leading edge , Successively released the industry's first general heterogeneous parameter server architecture 、4D Hybrid parallel training strategy 、 Many bright technologies such as end-to-end adaptive distributed training architecture , And fully Polish according to different model structures and sparse and dense characteristics , Supportable including computer vision 、 natural language processing 、 Personalized recommendation 、 Different algorithms in a wide range of fields, including scientific computing, achieve high-performance training on heterogeneous hardware , Effectively help the rapid iteration of large model technology innovation exploration .

The leading distributed technology and high-performance training features of the propeller , It supports the software and hardware scheme based on the propeller in MLPerf Continue to achieve excellent performance on , It supports the release of many industry-leading Wenxin large models , For example, the world's first knowledge enhancement model of 100 billion “ Pengcheng - Baidu · Literary heart ”, Knowledge enhanced power industry NLP Big model “ State Grid - Baidu · Literary heart ”, Knowledge enhanced financial industry NLP Big model “ PUFA - Baidu · Literary heart ”, And domestic hardware clusters AlphaFold2 Ten million level protein structure analysis model .

Conclusion

Where are the oars MLPerf Training v2.0 Got BERT Model training performance is the world's first eye-catching achievement . This is not only due to the long-term efforts of the propeller frame in the field of performance optimization , It cannot be separated from the help of hardware ecology . In recent years , The technical strength of the propeller is deeply recognized by the majority of hardware manufacturers , Cooperation is getting closer , Integrated software and hardware for coordinated development , Ecological co creation is fruitful . Not long ago （5 month 26 Japan ）,NVIDIA Launched in cooperation with the propeller NGC-Paddle The official launch . At the same time MLPerf In the list ,Graphcore Also achieved excellent results by using the propeller frame . future , The propeller will continue to create performance advantages , Continuous technological innovation in software and hardware collaborative performance optimization and large-scale distributed training , For the majority of users to provide more convenient 、 Easy to use 、 Deep learning framework with excellent performance .

MLPerf Introduce

MLPerf By AI Benchmark in the field of artificial intelligence initiated by world-renowned academic researchers and industry experts .MLPerf It aims to provide a fair 、 A practical benchmark platform , Show industry-leading AI The best performance of software and hardware system , The test results have been obtained AI General recognition in the field . Almost all mainstream hardware manufacturers and software service providers in the world will refer to MLPerf Build your own benchmark system based on the published results , To test the new AI Accelerating chip and deep learning framework in MLPerf Performance on the model .

Live broadcast announcement

7 month 6 Japan （ Wednesday ）20:00, Yu dianhai, the chief architect of flying oars, and Zeng Jinle, the senior R & D Engineer of flying oars, will broadcast live , Uncover secrets for everyone GPU Under configuration , Baidu PaddlePaddle performance 「 World first 」 The key technology behind it .

Official account of flying propeller , The background to reply 【 Study 】 Sign up , There are more gifts waiting for you in the live studio ！

Focus on 【 Flying propeller PaddlePaddle】 official account
Get more technical content ~