Pytorch Foundation

pytorch yes Facebook Developed deep learning library . Details visible Official website .

install

For the convenience of management , Usually in Anaconda or miniconda Create a new virtual environment installation in pytorch.

New virtual environment （MacOS）

Method 1 ： direct Anaconda Navigator Install in

stay Navigator>Environments Create a new environment in

Method 2 ： Create... In the terminal

Open the terminal and input the following command ：

#  View the installed environment 
conda env list

#  install pytorch A virtual environment 
conda create -n pytorch python==3.7

Other commands related to virtual environment operation ：

#  Delete the specified virtual environment 
conda remove -n pytorch --all

#  Activate pytorch A virtual environment 
conda activate pytorch

#  Exit virtual environment 
conda deactivate

install pytorch

mac It can be installed directly pytorch, The installation order can be found on the official website .

Check whether the installation is successful

import torch
torch.cuda.is_available()

The return value is False Then for CPU edition .

Basic concepts

Tensor

tensor （Tensor） It's a vector 、 Matrix direction n Expansion of dimension , yes pytorch Data storage in 、 The main tool of transformation .

establish Tensor

function	give an example	explain
tensor(data)	x = torch.tensor([1,2])	Generate a 1 D tensor [1,2]
empty(data)	x = torch.empty(3,2)	Generate a 3 That's ok 2 Column space tensor
ones(sizes)	x = torch.ones(3,2)	$$x=\left[\begin{array}{cc}1& 1\\ 1& 1\\ 1& 1\end{array}\right]$$
zeros(sizes)	x = torch.zeros(3,2)	$$x=\left[\begin{array}{cc}0& 0\\ 0& 0\\ 0& 0\end{array}\right]$$
eye(sizes)	x = torch.eye(3,2)	$$x=\left[\begin{array}{cc}1& 0\\ 0& 1\\ 0& 0\end{array}\right]$$
rand(sizes)	x = torch.rand(3,2)	Random generation obeys [0,1) Evenly distributed 3 That's ok 2 Column tensor
randn(sizes)	x = torch.randn(3,2)	Random generation obeys N(0,1) Normal distribution 3 That's ok 2 Column tensor
randperm(m)	x = torch.randperm(3)	Random 1 dimension [0,6) Integers are arranged randomly

operation

operation	explain	function
Add	x+y	z = x + y
		z = torch.add(x,y)
		w = torch.empty(0) torch.add(x,y,out =w)
		y.add_(x)
Subtraction	x-y	z = x - y
		z = torch.sub(x,y)
Multiplication	Multiply the corresponding position elements	z = x * y
		z = torch.mul(x,y)
division	The corresponding position element is divided by	z = x / y
		z = torch.div(x,y)
Matrix cross multiplication	Cross riding	z = x @ y
		z = torch.mm(x,y)
General multiplication	x,y Are all 1 dimension ： Point multiplication x,y Are all 2 dimension ： Differential multiplication x,y There is 1 A for 1 Another dimension is greater than 1 position ： Trigger broadcast mechanism	z = torch.matmul(x,y)

notes ： Broadcast mechanism For when two shapes 不 Identical Tensor When operating by element , First copy the elements appropriately so that the two Tensor After the shape is the same, operate according to the element .

Other operating

function	give an example	explain
view(shape)	x.view(2,3)	Turn the original 3 That's ok 2 The tensor of a column x Change it to 2 That's ok 3 Column
item()	x[0,0].item()	return x The first 1 That's ok 1 Column element values , Not Tensor Format

autograd

autograd yes pytorch Chinese vs Tensor A package that performs automatic derivation . When Tensor.requires_grad=True when , Will automatically track the Tensor All operations . The default is False.

Basic application process

#  Generate a tracking tensor 
x = torch.ones(2, 2, requires_grad=True)
y = x**2
out = y.sum()
#  Back propagation 
out.backward()
#  Output derivative 
print(x.grad)
#  New operation 
out2 = x.sum()
out2.backward()
print(x.grad) # Will accumulate out.backward() result 
#  Gradient clear 
out3 = x.sum()
x.grad.data.zero_()
out3.backward()
print(x.grad)

Block tracing

• use with torch.no_grad()

print(x.requires_grad)
print((x ** 2).requires_grad)

with torch.no_grad():
    print((x ** 2).requires_grad)

• use Tensor.data Carry out operations

x = torch.ones(1,requires_grad=True)
print(x.data) #  Or a tensor
print(x.data.requires_grad) # return False
y = 2 * x
x.data *= 100 #  Only changed the value , It won't be recorded on the calculation chart , So it doesn't affect gradient propagation 
y.backward()
print(x.grad)

Parallel computing

It is said that 1.12 Version of pytorch Will support mac GPU edition .

Network partitioning

Divide the model into several parts , Different parts are put in different places GPU

Layer-wise partitioning

Continue to divide the same part of the model into different tasks , In different GPU Calculate at the same time

Data parallelism（ Main stream ）

Split data , In different GPU Run different data on the same model

Reference resources

datawhale： Explain profound theories in simple language pytorch