Understanding of L1 regularization and L2 regularization [easy to understand]

Hello everyone , I meet you again , I'm your friend, Quan Jun .

One 、 Okam razor (Occam’s razor) principle ：

In all possible models , We should choose data that can well explain , And a very simple model . From a Bayesian point of view , The regular term corresponds to the prior probability of the model . It can be assumed that the complex model has a small a priori probability , A simple model has a large a priori probability .

Two 、 Regularization term

2.1、 What is regularization ？

Regularization is the realization of structural risk minimization strategy , Add a regular term or penalty term to the empirical risk , There are two kinds of regular terms L1 Regularization and L2 Regularization , perhaps L1 Norm sum L2 norm . For linear regression model , Use L1 The regularized model is called Lasso Return to ; Use L2 The regularized model is called Ridge Return to ( Ridge return )

2.2、 The relationship between regularization term and model complexity

The regularization term is generally a monotonically increasing function of model complexity , The more complex the model , The larger the regularization value is .

Generally speaking , Supervised learning can be seen as minimizing the following objective function ：

The second in the above formula 1 Item is experience risk , namely Model f(x) About the average loss of training data set ; The first 2 The term is a regularization term , Go to Constraining our model is simpler

3、 ... and 、L1 norm

3.1 Concept ： L1 Norm is the sum of the absolute values of the elements in a vector .

3.2 Why? L1 The norm makes the weights sparse ？

Any regularization operator , If he were Wi=0 You can't be tiny , And can be decomposed into “ Sum up ” In the form of , Then the regularization operator can achieve sparse .

3.3 What are the benefits of sparse parameters ？

（1） feature selection （Feature Selection）

Parameter sparse regularization can realize automatic feature selection , In the process of Feature Engineering , Generally speaking ,xi Most of the elements of （ features ） And its label yi unconcerned . When we minimize the objective function , These irrelevant features are considered , Although the minimum training error can be obtained , But for new samples , This useless information is instead considered , It interferes with the prediction of the sample . Sparse regularization sets the weight of these useless features to 0, Get rid of these useless features .

（2） Interpretability

Set irrelevant features to 0, Models are easier to explain . for example ： The probability of suffering from a certain disease is y, The data we collected x yes 1000 Dimensional , Our task is to find this 1000 How do the three factors affect the probability of suffering from this disease . hypothesis , We have a regression model ：y=w1*x1+w2*x2+…+w1000*x1000+b, Through the study , We finally learned w* There are only a few non-zero elements . For example, only 5 A non-zero w*, So this 5 individual w* Contains key information about the disease . in other words , Whether suffering from this disease and this 5 Two features are related , That thing has become much easier to deal with .

Four 、L2 norm

4.1 Concept ：L2 Norm refers to the sum of the squares of the elements of a vector, and then find the square root .

The regularization term can take different forms . For the regression problem , The loss function is the square loss , The regularization term is a parameter vector L2 The norm of .

4.2 Why? L2 Norms prevent over fitting ？

The left one ： Under fitting ; middle ： Normal fit ; On the right side ： Over fitting

Linear regression fitting diagram

Give Way L2 Regular term of norm ||W||2 Minimum , You can make W Each of the elements is very small , Are close to 0.（L1 Norm let W be equal to 0）, The smaller the parameter, the simpler the model is , The simpler the model, the less likely it is to produce over fitting .（ According to the linear regression fitting diagram above , Some parameters are limited to very small , In fact, it limits the influence of some components of the polynomial to be very small , This is equivalent to reducing the number of variables ）

Publisher ： Full stack programmer stack length , Reprint please indicate the source ：https://javaforall.cn/128182.html Link to the original text ：https://javaforall.cn