Loss function of depth model

Recently, I did a small project about location prediction . Because it involves regression , I used it huber Loss function , Now let's summarize the regression loss function , In order to deepen the understanding of

1. MSE and MAE The differences and advantages and disadvantages of

（1） Simply speaking ,MSE Simple calculation , but MAE Better robustness to outliers . If the training data is contaminated by outliers , that MAE Loss is better used . For example, see link 1, It can also be understood intuitively ：MSE Equivalent to the average ,MAE Equivalent to taking the median .
（2） However MAE There is a serious problem （ Especially for neural networks ）： The updated gradient is always the same , in other words , Even for small loss values , The gradient is also large . This is not conducive to model learning . To solve this defect , We can use varying learning rates , Reduce the learning rate when the loss is close to the minimum .

2. Better loss function

huber Loss function , see link 2

3. Dealing with imbalances Focal loss Loss function

Alleviate the imbalance between positive and negative samples . Be careful ：keras There is a problem of category imbalance in , You can train by adding class weights , I don't know and Focal loss Does it matter

4. Cross entropy loss function

Basic knowledge can be seen link 4.1
A previous interview question , Why use the cross entropy loss function ？ See link 4.2

5. Loss function in twin networks