The training set Loss converges, but the test set Loss oscillates violently?

问题场景

Today, when debugging the model, I found that it is on the training setLoss已经收敛了,但是在验证集上LossVibration is more severe,如下图所示：

Loss曲线

原因分析

Found it after checking various blogs online验证集Loss震荡的原因可能有如下：

• 数据问题,For example, the training set and the validation set are too different,数据量太小;
• batchsize太小,The rules of model learning are not enough“普适”;
• lossThe function is not suitable;
• 学习率太大,The model gets stuck in a local optimum;
• There is a problem with the network structure of the model;
• ......

After knowing the reason,can be checked one by one.

• 数据上,Check your own datasettrain和valid的划分情况,Basically the distribution of the data should be about the same;数据量上1w+的图像,应该也还行;
• lossThe function is changed to other tests and the effect is still the same;
• The learning rate uses a dynamic adjustment strategy,应该没什么问题（In the later tests, even if the initial learning rate is adjusted,The end result is still similar）;
• The model uses a more classic super-score model,应该问题不大;
• 调整了下bs,从32调整到48,The vibration was found to be smaller,效果如下图：

bs=32

bs=48

So the final analysis should bebatchsize太小的原因,If the point estimation can be increased, the effect will be better,但是奈何GPUNot enough memory.

总结

如果遇到Losshas converged on the training set,However, the shock on the validation set is more severe,Analyze the possible causes one by one,and try it.When trying, you should also pay attention to the theoretical analysis before running the model,Otherwise, it may be a waste of computing power.

参考

【1】https://blog.csdn.net/qq_40689236/article/details/106794155
【2】https://zhuanlan.zhihu.com/p/483488388