Chapter 5 decision tree and random forest practice

1 Over fitting problem of decision tree

1.1 Problem description

Decision tree has good classification ability for training , But the unknown test data may not have good classification ability , Weak generalization ability , That is, fitting phenomenon may have occurred .

1.2 resolvent

（1） prune

 

 

（2） Reasonable and effective sampling

bagging:

 OOB data

  Random forests

• Random forests /Bagging Relationship with decision tree

Decision tree is the basic classifier ;

SVM、Logistic Regression and other classifiers “ Total classifier ”, It's called random forest . 

give an example ： The return question

2 Return to

2.1 The algorithm process

do 100 Time bootstrap, Every time I get data Di（Di The length of is N）. For each Di, Use local regression （LOESS） Fit a curve . Then average these curves , Get the final fitting curve , The curve over fitting is weakened .

2.2 give an example

vote ：（1） Simple voting mechanism ： One vote against 、 The minority is subordinate to the majority 、 Threshold voting （2） Bayesian voting mechanism

Movie reviews ： bring As big as possible .

 

3 The use of random forests

3.1 Use random forest to calculate the similarity between samples

principle ： If two samples appear at the same leaf node at the same time, the more times , The more similar the two .

The algorithm process ： Record the number of samples as N, initialization NXN The zero matrices of S,S[i,j] Presentation sample i and j The similarity . about m A random forest formed by a decision tree , Traverse all leaf nodes of all decision trees （ sample i,j Appear at the same node , be s[i,j] Add 1）. End of traversal ,S Is the similarity matrix between samples .

3.2 Use random forests to calculate the importance of features

（1） Calculate the node through which the positive example passes , Use the number of passing nodes 、gini Coefficient and other indicators to judge the importance of characteristics .

（2） Randomly replace a column of data , Rebuild the decision tree , Calculate the change of the accuracy of the new model to judge the importance of the characteristics of this column .

3.3 Isolated forests

Isolated forests （Isolation Forest） Detect outliers by isolating sample points .

features 、 The dividing points are randomly selected , Then generate a certain depth of decision tree iTree, Several trees iTree form iForest.

To calculate iTree The length of the sample from root to leaf f(x), And then calculate iForest in f(x) The sum of F(x).

Test standard ：F(x) Smaller samples x Is an outlier .

summary

Decision tree / The code of random forest is clear 、 The logic is simple , While being competent for classification problems , It can also be used as the primary algorithm to explore data distribution .

The integration idea of random forest can also be used in the design of other classifiers .