Ml self study notes 5

Integrated algorithm and random forest

The integration algorithm

The integration algorithm is generally done with the tree model .
Bagging Is to train multiple tree models , Each tree model predicts a result , Sum the results of all tree models and then find the mean , A typical example is random forest , I'll talk about it later .
Boosting（ Lifting algorithm ）, Is to add a tree every time , Finally, the mean value is not sought

Random forests

brief introduction

1. Random ： Random sampling of data , For example, establish a random selection of each tree 80% The sample of ; Feature random sampling ： Each tree is randomly selected 80% Sample characteristics of （ Then the comparison column can be specified ）;
2. The forest ： Multiple trees , Sum the results of each tree and find the mean .
   

Random forest advantages and

give Negev The important characteristics can be judged by the process in the figure below . First, find out the error rate of original feature modeling , Modify the characteristic value of the feature you want to judge （ Disrupt or change the value ） Then the other features remain unchanged, and then modeling and calculating the error rate ; The two results approximate that this feature is not important , The second result is much larger than the first result, indicating that this feature is very important . The importance can also be judged by the position of the feature in the tree , The closer to the root node, the more important （ It is obtained directly by algorithm ）.

The more tree models, the better , After reaching a certain number, the accuracy rate is almost floating up and down .

Lifting algorithm （Boosting） summary

Introduction to the algorithm

Suppose the first tree predicts 950（ The label is 1000）, that B Just upgrade A Of ,B The prediction is not 1000 了 , Is to make up A Not done well ,B The goal is 1000-950=50, hypothesis B Predicted 30, turn C when ,C The tree regards the front as a whole , The prediction result is expected 1000-950-30=20, hypothesis C forecast 18,D It's also , The prediction label is 2.

Lifting algorithm formula

The previous part is the prediction result of the previous tree , The following part is the current tree model , Previous results F_m-1(x_i) Add the result of the current tree h(x_i) And the original result y_i Compare , If improved, add , If there is no improvement, don't add . Guaranteed to join a tree , It will be stronger than before .

The model represents

1. AdaBoost: If the first division is the result of Figure 1 , There is a wrong triangle division , Then the next round will increase the weight of this triangle
2. Xgboost The algorithm is mentioned before ABCD How did the tree ascend .

Stacking model （stacking）

Model overview

Use multiple classifiers , such as LR（ Logical regression ）,DT（ Decision tree ）,RF（ Random forests ）, Make separate predictions , Then take the result as input and operate with a model .