Over fitting and regularization

Over fitting

This is a neural network training process , The problems we often encounter , Simply speaking , Is the performance of the model , Learning ability is too strong , So that Training set All the details of have been recorded . When you meet Test set , It's when I haven't seen data before , There will be obvious mistakes .

The reasons causing

The most essential reason is ： Too many parameters （ The model is too complex ）
Other reasons are ：

1. The distribution of test set and training set is different
2. The number of training sets is too small

terms of settlement

For the above reasons , Several countermeasures can be put forward

1. Reduce model complexity , Regularization is commonly used .
2. Enhanced training set

norm Norm（Minkowski distance ）

Definition

A norm is a function , It gives each vector in a vector space a length or size .

For the zero vector , The length of 0.
$$\Vert x{\Vert }_p={\left(\sum _{i=1}^n|x{|}^p\right)}^{\frac{1}{p}}$$

Properties of norms

• Nonnegativity $\Vert x\Vert \ge 0$
• Homogeneity $\Vert cx\Vert =|c|\Vert x\Vert$
• Trigonometric inequality $\Vert x+y\Vert \le \Vert x\Vert +\Vert y\Vert$

Norm characteristic

• $L_0$ norm ： Number of non-zero elements
• $L_1$ norm ： The sum of absolute values
• $L_2$ norm ： Euler distance
• $L_{\infty }$ norm ： The absolute value Maximum The absolute value of the element of

Regularization

Objective function plus a norm , As a penalty . If a parameter is larger , It will increase the norm , That is, the penalty item increases . So under the action of norm , Many parameters are getting smaller .
The smaller the parameter , It shows that it plays a smaller role in neural network , That is, the smaller the impact on the final result , Therefore, it can make the model simpler , And it has more generalization ability .

Regularization is also a Superior bad discard Thought , Although many parameters are useful for the model , But in the end, only important parameters can be preserved （ It's worth more , Have a great impact on the results ）, And most parameters have been eliminated （ Small values , It has little effect on the results ）.