[AI system frontier dynamics, issue 40] Hinton: my deep learning career and research mind method; Google refutes rumors and gives up tensorflow; The apotheosis framework is officially open source

1、Geoffrey Hinton： My 50 years of in-depth study career and Research on mental skills

 

After suffering from wind and frost , This one has 74 Year old “ Deep learning Godfather ” Still fighting AI Research frontline , He is not afraid of questions from other scholars , Will also frankly admit those judgments and predictions that have not been realized . No matter what , He still believes , Ten years after the rise of deep learning , This technology will continue to release its energy , And he is also thinking and looking for the next breakthrough .

 

https://mp.weixin.qq.com/s/kRdlK3VEqeKSr9up0ay5dg

 

2、Google Refute a rumor and give up TensorFlow

 

expectation TensorFlow The future of ,Google Further shows the attitude ,「 We intend to continue to develop TensorFlow, As an application ML First class platform , And JAX side by side , Push ML The scope of the study . We will continue to invest in these two ML frame , To promote the research and application of millions of users .」

 

https://mp.weixin.qq.com/s/JAGHRVUb1Mla_wIWoyJPeA

 

3、 The apotheosis framework is officially open source , Easy pre training and fine tuning “ Granting titles to gods ” Major models

 

FengShen Is for “ Granting titles to gods ” A pre training model framework tailored to a series of large models . Fengshen list team focuses on Chinese NLP Big model open source , However, the increase of model brings more than the increase of training difficulty , The difficulty of use has also increased . With FengShen, Users can use their own needs , from “ Granting titles to gods ” Select the pre training model , recycling FengShen Quickly fine tune downstream tasks .

 

https://mp.weixin.qq.com/s/NtaEVMdTxzTJfVr-uQ419Q

 

 

4、 Training GPT-3, Why can't the original in-depth learning framework ？

 

This paper will focus on the technical challenges faced by deep learning framework in supporting large-scale pre training model , And the basic solutions of the current various frameworks , Help algorithm engineers have a clearer understanding of the distributed training capabilities of various frameworks in the industry .

 

https://mp.weixin.qq.com/s/qZ6qYfAX442vQBiJXwt6uA

 

5、 Combat software system complexity ①： If not necessary , Do not add entities.

 

We often face the problem of how to evaluate the quality of a large software system . The primary evaluation index must be function , Whether the software meets the main requirements (do right things). If there are multiple technology paths that can achieve the same function , People tend to choose simpler methods . Occam razor guidelines “ If not necessary , Do not add entities. ” This preference is very well summarized , The preference for simplicity is to fight the challenge of complexity , The underlying logic is ：“ Simple is the right way ”（do things right）.

 

https://mp.weixin.qq.com/s/TmbTQYakDcDEh7nbnfumPQ

 

 

6、 How to develop machine learning system ： High performance GPU Matrix multiplication

 

Modern machine learning frameworks will put users Python The program is compiled into calls GPU Operator data flow program . Matrix multiplication as many GPU Basic operation of operator , How to optimize its performance has become the top priority . what's more , In the process of optimizing matrix multiplication , We will use many GPU High performance development skills . These skills will develop all kinds of GPU operator （ Convolution , Pool, etc ） Frequently used . therefore , By understanding high performance GPU Multiplication , master GPU Using skills will become major AI Important teaching contents of companies and colleges .

 

https://zhuanlan.zhihu.com/p/531498210

 

7、 For deep learning GPU share

 

GPU Sharing involves a wide range of technical aspects , Include GPU framework （ Calculation , Storage, etc ）,Cuda,IO（ Memory , memory ）, Machine learning framework （Tf,Pytorch）, colony & Dispatch ,ML/DL Algorithm characteristics , signal communication （ Within a single machine and between multiple machines ）, Reverse engineering and so on , It's a top-down job . This article hopes to provide a reference to GPU Sharing work sharing , I hope to discuss with researchers in related fields .

 

https://zhuanlan.zhihu.com/p/285994980

 

 

8、 Explain profound theories in simple language GPU Optimization series ：reduce Optimize

 

This article mainly introduces how to GPU Medium reduce Algorithm optimization . Currently for reduce The optimization of the ,Nvidia Official documents of reduce Optimize It has been said in detail , But it's too lean , Many things pass in one stroke . For newcomers in this field , It's still hard to understand . Therefore, on the basis of official documents , Make a more in-depth explanation and explanation , As far as possible, every reader can thoroughly understand through this article reduce Optimization technology .

 

https://zhuanlan.zhihu.com/p/426978026

 

9、 from MLPerf Talking about ： How to lead AI The next wave of accelerators

 

Maybe , If we leave the tensor intensive model, we can achieve “AI Speed up 2.0”, welcome AI The next wave of hardware . At that time ,“AI Speed up 2.0” There is no need to multiply the matrices , But it supports irregular computing mode , And it has flexible representation and arbitrary parallel computing engine . Accelerators with these characteristics （ Or heterogeneous integration of such accelerators ） It will have a wider range of applications , It is also closer to the real AI.

 

https://mp.weixin.qq.com/s/n116POs9H9v5wqJA7_km6A

 

10、 Open source compiler for Xiaobai

 

the-super-tiny-compiler This treasure level open source project , It's just 1000 Mini compiler around lines , Comments account for... Of the code 80%, The actual code is only 200 That's ok ！ The sparrow is small, but it has all five internal organs , Complete implementation of the basic functions required by the compiler , adopt Code + notes + Explain Let you get started with an open source project compiler .

https://zhuanlan.zhihu.com/p/515999515

 

 

11、OneFlow The source code parsing ： Automatic inference of operator signature

 

Deep learning framework is a complex system , And the operator that users use most is （op）. User pass op Build a model , Training 、 forecast . This note starts from op Starting with , Let's see Python Front end to C++ Bottom ,OneFlow How to execute the operator's computational logic .

 

https://mp.weixin.qq.com/s/_0w3qhIk2e8Dm9_csCfEcQ

 

Everyone else is watching

• OneFlow v0.7.0 Release
• The illustration OneFlow Learning rate adjustment strategy of
• The optimal parallel strategy of distributed matrix multiplication is derived by hand
• OneFlow The source code parsing ： Automatic inference of operator signature
• Train a large model with hundreds of billions of parameters , Four parallel strategies are indispensable
• Reading Pathways（ Two ）： The next step forward is OneFlow
• Hinton： My 50 years of in-depth study career and Research on mental skills

Welcome to experience OneFlow v0.7.0：https://github.com/Oneflow-Inc/oneflow/