PyTorch: In-place Operation

In-place Operation

In-place operation stay pytorch Middle refers to support in situ modification tensor, This can avoid new memory overhead , Mainly to reduce the consumption of video memory

But in pytorch When back propagating to get the calculation graph, you need to obtain the value of the intermediate variable , If you use in-place After the operation, the intermediate value is lost , Unable to get calculation diagram , But why are some activation functions written in-place Operation? ？

With ReLU Function as an example , It does not change when the input value is positive in-place Whether it does not affect the result ; When the input is negative, it is 0 No back propagation, so it doesn't affect

common In-place operation

1. Symbolic calculation

*=+= Wait for the operation

stay pytorch Try to avoid

tensor = torch.tensor([[1,2,3], [4,5,6]])
print(tensor.data_ptr())
tensor += 2
print(tensor.data_ptr())
tensor = tensor+2
print(tensor.data_ptr())
''' 140430616339712 140430616339712 # in-place 140430616321664 # out-of-place '''

（ Continue to supplement ）

common Out-of-place operation

1. Symbolic calculation

a=a+b etc. , Reference resources In-place operation