Deep Learning Theory - Overfitting, Underfitting, Regularization, Optimizers

 

 

Data augmentation: 1. Do not overdo it, otherwise it will only increase the training time and will not increase the generalization ability; 2.Add extraneous data

L2 regularity: tend to respond to the common characteristics of training set samples; make the model prefer samples with small parameters to reduce the risk of overfitting

 

 

 

 

 

 

 

Several common optimizers

For sparse data, try to choose an optimization method with an adaptive learning rate. It does not need to be adjusted manually. It is better to use the default value.

Stochastic gradient descent usually takes longer to train and is prone to saddle points, but results are more reliable with good initialization and learning rate scheduling.

Overall, Adam is by far the best choice.