Pytorch notes: validation, model eval V.S torch. no_ grad

1 validation The general framework of

The model is model, The optimizer is optimizer

min_val_loss = np.inf
for epoch in range(1, epochs + 1):
    
    ############################ Start of training section #############################
    model.train()
    train_losses = []
    for (batch_x, batch_y) in train_loader:

        output = model(batch_x) 
        loss = criterion(output, batch_y) 

        optimizer.zero_grad()
        loss.backward() 
        optimizer.step()
        #pytorch  The third one 

    ############################ End of training #############################


    ############################ The verification section begins #############################
    model.eval()
    for (batch_x, batch_y) in val_loader:

        with torch.no_grad():
            output = model(batch_x) 
            loss = criterion(output, batch_y)

        val_losses.append(loss.item())
    val_loss = np.mean(val_losses)

    if val_loss < min_val_loss:
        min_val_loss = val_loss
        torch.save(model.state_dict(), save_path)
        # Save the best model 
    ############################ End of validation section #############################

test When , You can load the parameters corresponding to the optimal model （model.load_state_dict）, To test

2 model.eval() V,S with torch.no_grad()

2.1 The same thing

stay PyTorch In the middle of validation when , Use them to switch to test mode .

eg, about dropout Layer and the batchnorm layer ：

• stay train In mode ,dropout The network layer will follow the set parameters p Set the probability of retaining the active unit （ Retention probability =p); batchnorm The layer will continue to compute the data mean and var And update .
• stay val In mode ,dropout The layer will let all the active units pass through , and batchnorm Layers stop calculating and updating mean and var, Use directly what you've learned in the training phase mean and var value .

2.2 Difference

• model.eval() Still involved gradient Calculation and storage of , And training Model as , It's just that there's no back propagation .

• **with torch.zero_grad()** Then stop autograd Module work , That is to say stop it gradient Calculation , In order to speed up and save video memory , Thus saving GPU Computing power and memory , But it won't affect dropout and batchnorm Layer behavior .（ These two are still train Mode ）

• ——> Use the two together