How to build a deep learning framework?

At present, the framework of deep learning is becoming more and more mature , For users, the degree of encapsulation is higher and higher , The advantage is that these frameworks can now be used as tools very quickly , Experiments can be carried out with very little code , The disadvantage is that the implementation behind the possibility is hidden . In this article, the author will take you to design and implement a lightweight ( about 200 That's ok )、 Easy to extend deep learning framework tinynn, I hope it will be helpful for you to understand the basic design and implementation of the deep learning framework .

This paper will first analyze the process of deep learning , Abstract the key components in neural network , Determine the basic framework ; Then code the components in the framework ; Finally, based on this framework, a MNIST Examples of classification .

Component abstraction

First, consider the flow of neural network operation , Neural network operation mainly includes training training And forecasting predict ( or inference) Two phases ,

The basic process of training is ： input data -> Network layer forward propagation -> Calculate the loss -> Network layer back propagation gradient -> Update parameters ;

The basic process of forecasting is input data -> Network layer forward propagation -> Output results .

From an operational point of view , There are three main types of calculations ：

Data flows directly at the network layer

Forward propagation and back propagation can be regarded as tensors Tensor( Multidimensional arrays ) Flow between network layers ( The forward propagating flow is the input and output , The of backward propagating flow is gradient ), Each network layer will perform certain operations , Then input the results to the next layer .

Calculate the loss

The intermediate process of connecting forward and backward propagation , Defines the difference between the output of the model and the real value , Used to provide the information needed for back propagation

Parameters are updated

A kind of calculation that uses the calculated gradient to update the network parameters

Based on these three types , We can make an abstraction of the basic components of the network

tensor tensor , This is the basic unit of data in neural network

layer The network layer , Be responsible for receiving the input from the upper layer , Perform the operation of this layer , Output the result to the next layer , because tensor There are two directions of flow: forward and reverse , Therefore, for each type of network layer, we need to implement forward and backward Two operations

loss Loss , After given the predicted value and real value of the model , The component outputs the loss value and the gradient about the last layer ( Used for gradient return )

optimizer Optimizer , Responsible for updating the parameters of the model using gradients

Then we need some components to put the above 4 Integrate... Basic components together , To form a pipeline

net The component is responsible for managing tensor stay layer Forward and backward propagation between , At the same time, it can provide acquisition parameters 、 Set parameters 、 Get the interface of gradient

model Component is responsible for integrating all components , The formation of the pipeline. namely net Component forward propagation -> loss Components calculate losses and gradients -> net The component will propagate the gradient back -> optimizer The component updates the gradient to the parameter .

Share some of my artificial intelligence learning materials for free , Including some AI Common framework actual combat video 、 Image recognition 、OpenCV、NLQ、 machine learning 、pytorch、 Computer vision 、 Videos such as deep learning and neural network 、 Courseware source code 、 Famous essence resources at home and abroad 、AI Hot papers 、 Industry reports, etc .

For better systematic learning AI, I recommend that you collect one .

Here are some screenshots , Click here to download the materials for free .

One 、 Artificial intelligence courses and projects

Two 、 Famous essence resources at home and abroad

3、 ... and 、 Collection of papers on artificial intelligence

Four 、 AI Industry Report

Learn Artificial Intelligence well , Read more , Do more , practice , If you want to improve your level , We must learn to settle down and learn slowly and systematically , Only in the end can we gain something .

Small partners in need , Click here to download the materials for free .