Pytorch framework

Article, understand PyTorch Internal mechanism - You know Link to the original text ：http://blog.ezyang.com/2019/05/pytorch-internals/ The source of the original text is ：https://archwalker.github.io Original address ： Article, understand PyTorch Internal mechanism translator's Preface ： This post is a very new introduction PyTorch The internal mechanism of the article , author Edw…https://zhuanlan.zhihu.com/p/338256656PyTorch – Internal Architecture Tour | Terra Incognitahttps://blog.christianperone.com/2018/03/pytorch-internal-architecture-tour/PyTorch structure 、 Architecture analysis _windrisess The blog of -CSDN Blog _pytorch framework pyTorch Architecture references ： Home page - PyTorch Chinese document (pytorch-cn.readthedocs.io) List of articles pyTorch framework torch What is it? pytorch Medium torchtorch.Tensortorch.Storagetorch.nn Contains multiple subclasses ： Containers (Containers): The network layer ： Function package ：torch.nn.functional Build a good network ：torch.autograd:torch.optim:torch.cuda:torch.utils:torch.utils.data.Datashttps://blog.csdn.net/qq_28726979/article/details/120690343 List Pytorch frame - You know Make a statement first , This blog is not an explanation Pytorch Internal details of the article , More is the macro explanation , I hope readers can stand on We can see clearly on the shoulders of our predecessors Pytorch Frame composition , At the same time, I hope readers can go further More about Pytorch 's blog post , Interested partners can pay attention to …https://zhuanlan.zhihu.com/p/334788042PyTorch JIT - You know If searching PyTorch JIT, What will be found will be 「TorchScript」 Documents , So what is JIT Well ？JIT and TorchScript What's the connection ？ The article will only focus on the conceptual part , If you focus on details or implementation parts , At the end of the article, there is a complete Demo For reference . what …https://zhuanlan.zhihu.com/p/370455320        pytorch The preview of the frame is in use numpy Write an in-depth learning framework for the accumulation of basic knowledge , and pytorch In contrast, this framework , And then up to mmcv This encapsulation of the training process , Finally, to the specific algorithm . But yes. pytorch The internal parsing of is very difficult , We mainly focus on its code structure ,torch The bottom is c/c++ Written ,pytorch yes torch Of python Interface .

         It's on it pytorch and torchvision Interface representation of ,pytorch The bottom layer of is all links torch._C, however torchvision Is based on torch Of , yes python Written ,torchvision and openmmlab It's done mm series ,detectron The series is almost , It's a pytorch A layer of encapsulated library is made on , To this extent, the algorithm model is basically used . In the framework of deep learning, although it can be based on numpy To write ,numpy The bottom layer is made of C/fortan, A lot of optimization has been done in matrix calculation , Soon , Want to have list turn array The thinking of , however numpy It can not fully support many deep learning operators and high-dimensional tensor.

1.torch

        torch Defined multidimensional tensor Structure and based on tensor A variety of operations , and numpy almost , But for deep learning .

1.1 torch.Tensor

        16 An integer torch.ShortTensor,32 An integer torch.IntTensor,64 An integer torch.LongTensor,16 Bit floating point type torch.HalfTensor,32 Bit floating point type torch.FloatTensor,64 Bit floating point type torch.DoubleTensor,torch.Tensor The default is torch.FloatTensor

        0.4 After the version ,Tensor and Variable The merger of , A variable is a tensor that can construct a computational graph and can be derived automatically .

1.2 torch.Storage

        be responsible for torch.Tensor The underlying data storage . Suppose a k Order tensor , Its dimension is （n1,n2,...,nk）, Because the computer memory is a continuous address space , So what is stored in the actual stored procedure is 1 Dimension vector .

1.3 torch.nn

        nn It is the core of neural network modularization

1.3.1 torch.nn.init 

        Defines the initialization of network weights , Many initialization methods are used ,torch.nn.init.uniform_,torch.nn.init.normal,torch.nn.init.kaiming_uniform...

1.3.2 torch.nn.functional

        Defined pytorch Function module in , yes def, No class, It mainly includes loss function (ctc_loss/nll_loss/poisson_nll_loss/kl_div/cross_entropy/binary_cross_entropy_with_logits/smooth_l1_loss/l1_loss/mse_loss/marhin_ranking_loss/hinge_emdedding_loss/multilabel_margin_loss/soft_margin_loss/multilabel_soft_margin_loss/cosine_embedding_loss/multi_margin_loss), Activation function (relu/glu/hardtanh/relu6/elu/selu/celu/leaky_relu/prelu/rrelu/gelu/hardshrink/tanhshrink/softsign/softmin/softmax/gumbel_softmax/log_softmax/tanh/sigmoid), Pooling (avg/max_pool1d/2d/3d,max_unpool1d/2d/3d,adaptive_avg_pool2d), Convolution (conv1d/2d/3d), interpolation (upsample/interpolate/upsample_nearest/upsample_biliinear/grid_sample/affine_grid/pad), normalization (batch_norm/instance_norm/layer_norm/group_norm/local_response_norm)

1.3.3 torch.nn.modules

        Definition torch Network layer and container in (container), The network mainly includes the class form of the above function ,forward Is called in functional The function in , Functions are mainly encapsulated in c++ Medium ,rnn and transformer This basic network component , And the base classes of all networks Module, Inherited from Module Of Sequential,ModuleList etc. .

1.4 torch.autograd

        Automatic differentiation module ,torch.autograd.backwar Back propagation after finding the loss function ,torch.autograd.grad For deriving one scalar from another tensor .

1.5 torch.optim        

  Define a series of optimizers (Adadelta/Adagrad/Adam/AdamW/SparseAdam/Adamax/ASGD/SGD/Rprop/RMSprop/Optimizer/LBFGS), as well as lr_scheduler, A lot of definitions lr Optimization method .

1.6 torch.cuda

        torch.cuda The module defines and cuda A series of functions related to operations , Including inspection cuda Is it available , Corresponding to the current process gpu Serial number , eliminate gpu Cache on , Set up gpu Computing flow on , Sync gpu All kernel functions executed on .

1.7 torch.distributed 

        Distributed computing module , Provide pytorch Parallel running environment , The main supported backend is MPI,GLoo,NCCL etc. .pytorch The distributed working principle of is to start multiple parallel processes , Each process has a backup of the model , Then input different training data to multiple parallel processes , Calculate the loss function , Each process does back propagation independently , Finally, the gradient of the weight tensor of all processes is reduced , The back end is mainly used for data broadcasting and data collection , The former is to transfer data from one node （ process ） Propagate to another node （ process ）.pytorch The distributed computing module not only provides a wrapper for the back end , It also provides some startup methods to start multiple processes .

1.8 torch.jit

         torch.jit yes pytorch Real time compiler for , hold pytorch Dynamic graph can be transformed into static graph that can be optimized and serialized .keras Medium model.compile and tf Medium graph All are jit Example .pytorch Of jit The realization is called torchscript,jit yes python and c++ The bridge , We can use python Training models , And then through jit Turn the model into a language independent module (onnx).jit Two modes are supported ：Scripting： You can use it directly TorchScript Language To define a PyTorch JIT Module, And then use  torch.jit.script  To turn him into TorchScript Module And save it as a file . and TorchScript Language Itself is also Python Code , So it can be written directly in Python In file . Use TorchScript Language It's like using TensorFlow equally , Define a complete diagram before you need to . about TensorFlow We know we can't use it directly Python Medium if Wait for statements to do conditional control , It needs to use tf.cond, But for the TorchScript We can still use it directly if and for Wait for conditional control statements , So even on static graphs ,PyTorch Still adhering to 「 Easy to use 」 Characteristics of .TorchScript Language It's static Python A subset of , Static types are also used Python 3 Of typing Module to achieve , So write TorchScript Language The experience is also similar to Python As like as two peas , Just some Python Feature cannot be used （ Because it's a subset ）, Can pass  TorchScript Language Reference  To see and native Python Similarities and differences .Tracing： Use TorchScript Module A simpler way to do this is to use Tracing,Tracing You can directly PyTorch Model （torch.nn.Module） convert to TorchScript Module.「 track 」 seeing the name of a thing one thinks of its function , Just need to provide a 「 Input 」 To make the model forward Again , To the flow path through this input , Get the structure of the graph . This way is good for forward It is very practical for a model with simple logic , But if forward There are many process control statements in it , There may be problems , Because the same input cannot traverse all logical branches .

1.9 torch.multiprocessing

        Defined pytorch Multiple processes in api, You can start different processes , Each process runs a different model , And can share tensors in process pieces , The shared tensor can be expressed in cpu On , It can also be in gpu On , Multi process api It also provides python Native multiprocess api The same set of functions , Include lock&queue etc. .

1.10 torch.onnx

        Defines export and load onnx Function of .

2.torch.utils

        Here's some python Auxiliary function of .

2.1 torch.utils.data

        Introduced Dataset and DataLoader,Dataset A dataset that contains all the data , A specific piece of data can be obtained by indexing ,DataLoader Through the packaging of data sets , Get an iterator , Generate bs data .

2.2 torch.utils.tensorboard

        This piece is commonly used tensorboardx There are many substitutes .

3.torchvision

        torchvision Equivalent to the torch Made a visual version of the algorithm library , This and mm Some algorithm libraries in the series are similar . Generally in DataLoader When using data processing transforms, This function is in torchvision Medium , Otherwise, there is no need to import torchvision.