RF, gbdt, xgboost feature selection methods "recommended collection"

Hello everyone , I meet you again , I'm your friend, Quan Jun .

RF、GBDT、XGboost Can do feature selection , It belongs to the embedded method in feature selection . For example sklearn in , You can use attributes feature_importances_ To see the importance of features , such as ：

from sklearn import ensemble
#grd = ensemble.GradientBoostingClassifier(n_estimators=30)
grd = ensemble.RandomForestClassifier(n_estimators=30)
grd.fit(X_train,y_train)
grd.feature_importances_

Copy

But how do these three classifiers calculate the importance of features ？ Let's explain it separately .

1. Random forests （Random Forest）

Use data outside the bag (OOB) Make predictions . Random forest in each re sampling to establish decision tree , There will be some samples that are not selected , Then these samples can be used for cross validation , This is also one of the advantages of random forest . It can avoid cross validation , Direct use oob _score_ To evaluate the performance of the model .

The specific method is ：

1. For every decision tree , use OOB Calculate the data error outside the bag , Write it down as errOOB1;

2. Then randomly pair OOB The characteristics of all samples i Add noise interference , Calculate the data error outside the bag again , Write it down as errOOB2;

3. Suppose there is N tree , features i The importance of sum(errOOB2-errOOB1)/N;

If random noise is added , The accuracy of out of pocket data has decreased significantly , It shows that this feature has a great impact on the prediction results , Then it shows that its importance is relatively high

2. Gradient lifting tree （GBDT）

Mainly by calculating the characteristics i The average value of importance in a single tree , The calculation formula is as follows ：

among ,M Is the number of trees . features i The importance of a single tree is mainly calculated according to this feature i The reduction in loss after splitting

among ,L Is the number of leaf nodes ,L-1 Is the number of non leaf nodes .

3. XGboost

XGboost It is calculated by the sum of the number of splits in each tree , For example, this feature splits in the first tree 1 Time , Second tree 2 Time ……, Then the score of this feature is (1+2+…).

Publisher ： Full stack programmer stack length , Reprint please indicate the source ：https://javaforall.cn/127541.html Link to the original text ：https://javaforall.cn