Machine learning by Li Hongyi 2. Regression

One 、 Return to （Regression）

Return to （Regression）： Find a function Function, By entering a feature $x$, Output a value $Scalar$ .

Application, for example,

• Stock market forecast （Stock market forecast）

• Autopilot （Self-driving Car）

• recommendation （Recommendation）

• Pokemon Elf attack power prediction （Combat Power of a pokemon）：

Two 、 Model steps

2.1 The model assumes - Linear model

• Univariate linear model （ A single feature ）

Model to represent ：$y=b+wx$

• Multivariate linear model （ Multiple features ）

Model to represent ：$y=b+\sum w_ix_i$

• $x_i$： All kinds of characteristics (fetrure)
• $w_i$： The weight of each feature
• b： Offset

2.2 Model to evaluate - Loss function

A single feature ：$x_{cp}$.

Definition $x^1$ It's pre evolutionary CP value , $\hat{y}^1$ For the evolved CP value ,$\hat{}$ What it represents is the real value .

collect 10 Group true value , With these real data , How do we measure the quality of the model ？ Mathematically speaking , We use distance . seek 【 After evolution CP value 】 And 【 The model predicted CP value 】 Bad , To determine the quality of the model . That is, using the loss function （Loss function） To measure the quality of the model .

$$L(f)=\sum_{n=1}^{10}(\hat{y}^n-f(x_{cp}^n))^2 \to L(w,b)=\sum_{n=1}^{10}(\hat{y}^n-(b+w\cdot x_{cp}^n)^2$$

▲ Loss function （Loss Function）

take $w$ and $b$ Show... In two-dimensional coordinates

• Each point in the figure represents the corresponding point of a model $w$ and $b$;
• The darker the color, the better the model .

▲ w and b Show... In two-dimensional coordinates

2.3 Model optimization - gradient descent

A single feature ：$x_{cp}$.

How to select the optimal model （ Find out what makes Loss Function The smallest $w$ and $b$）

▲ Definition f*

• From the simplest, there is only one parameter $w$ Starting with , Definition $w^* = arg\ \underset{x}{\operatorname{\min}} L(w)$

step 1： Choose one at random $w^0$

step 2： Calculate the differential , That is, the current slope , Determine the direction of movement according to the slope

• Greater than 0 To the right （ increase ww）
• Less than 0 Move to the left （ Reduce ww）

step 3： Move according to the learning rate

Repeat step 2 And steps 3, Until you find the lowest point

▲ Gradient descent process

• For two parameters $w$ and $b$, The procedure is similar to one of the above parameters , What we need to do is partial differential .

▲ Two parameter partial differential process

The challenge of gradient descent algorithm in the real world

• problem 1： The best at the moment （Stuck at local minima）
• problem 2： be equal to 0（Stuck at saddle point）
• problem 3： Tend to be 0（Very slow at the plateau）

▲ The problem of gradient descent

In the linear model, it is the shape of a bowl （ Valley shape ）, Gradient descent can basically find the best , But in other more complex models , Will meet problem 2 and problem 3 .

Verify whether the model is good or bad

Use Average error of training set and test set To verify the quality of the model .

3、 ... and 、 Over fitting （Overfitting）

Based on a simple model , It can be optimized , Choose a more complex model （ One yuan N Sublinear model ）, Or use polynomial Fitting .

If we choose a higher power model , Better models in the training set , On the test set, the effect may be worse . This is the problem of over fitting the model on the training set .

▲ Over fitting （Overfitting） The problem of

Four 、 Regularization （Regularization）

For more features , But the weight $w$ The weight of some features may be too high , Still lead to overfitting, You can add regularization .

$$Loss function :L=\sum_{n}(\hat{y}^n-(b+\sum w_ix_i)^2\\[2ex] The regularization term is introduced \to L=\sum_{n}(\hat{y}^n-(b+\sum w_ix_i)^2+\lambda\sum (w_i)^2 \\[2ex] \lambda\sum (w_i)^2： Regularization term .$$

▲ Regularization （Regularization）

▲ Adjust the λ Get the best model

5、 ... and 、 summary

Datawhale Team learning , Li Hongyi 《 machine learning 》Task2. Regression（ Return to ）, It mainly includes the definition of regression 、 To create a model 、 How to optimize the model 、 Possible problems in the process of optimizing the model and using regularization to solve the problem of over fitting .

——END——