Pytorch training CPU usage continues to grow (Bug)

CPU Memory continues to increase during training , rise , Use gc.collect() Invalid

1. Use

@profile  You can see cpu Occupancy 

2. Check for mistakes

python Content and tensor Whether there is a direct exchange of content, focus on checking , Focus on the accumulated places , This will lead to memory leakage

Example

such as acc += list(correctness) To the left of the equal sign is python Native list,correctness yes torch.FloatTensor

Switch to acc += list(np.array(correctness.cpu()))  

reason ：list And tensor There is a memory leak problem in mutual transfer , If you need to switch , Should be tensor from cuda-->cpu-->numpy.array

Example

total_loss +=loss1------->total_loss +=loss1.item()  (.item() Is to get the element value in an element tensor )

Forget when reasoning torch.no_grad() Cause gradient accumulation