[feature selection] several methods of feature selection

1. 1. feature selection *

Invalid variable

Irrelevant variables , Redundant variables

Feature selection of statistical methods

Variance thresholding 、 Chi square test 、ANOVA Inspection and T test 、 Pearson correlation coefficient

Selection of highly relevant features （ Redundant variables ）

Feature selection of model mode

Decision tree 、 Logical regression , Random forests ,XGBoost

The model will automatically select variables

Recursive feature selection .

Slowly eliminate the features , Limit to a specific range .

 

When input increases , Data must be added , Otherwise, the model will be unstable ,

1. 1. 1. Invalid variable

Irrelevant variables , Redundant variables

 

Redundancy： The correlation between the two variables is too high , explain 1 Whether the concepts of the two may be close , That is, redundant variables , You can adopt the method of merging . Even delete fields , Both bring information

Irrelevancy：X4,X3 Is irrelevant variables ,X4 When it gets larger, you will find the change of the target value . When X3 The predicted value is random when it changes , Unrelated , Unable to bring information .

 

1. 1. 1. Feature selection of statistical methods

VT Variance thresholding ： Calculate the variance of numeric fields , If below a certain value , It means that it contains insufficient information .

Variance cannot be standardized in advance . such as Z-scold Its variance is 1, The mean for 0

A threshold must be determined , Delete this field

Binary variable ： Code one of them as 1, One code is 0 The variance is P（1-P）（ First do feature transformation ）

 

When the variance is larger , Description is the more important field . The maximum is 0.25.

Of course , This has nothing to do with the goal

1. 1. 1. Statistical inspection method ：

The relationship between the input field and the target field

Category field ： Chi square test ： The relationship between the input field and the target field

Numeric fields ：ANOVA test （ The target field is greater than 2 Just go ）：T test （ The target field has only 2 It's worth , such as yes or no）： To verify the relevance between the input field and the target field .

ANOVA Case study ： Whether background music will affect consumers' mood . music （ Input field ） Relationship with alcohol purchase .

No music ,French Accordion ,italian Accordion

Alcohol ：French、italian、 Other alcoholic beverages

statistic

 

Real sales minus the sum of expectations divided by the sum of expectations

 

 

This is the expected frequency . Let the two be independent , probability 1 Multiply by the probability 2, Multiply by total 243.

Subtract the following table from the above table , Sum of squares , Divide by the sum of the mean

 

The larger the value, the better . The value of comparison can be found in the table ,

First calculate its chi square value , Use this value to look up the table , The corresponding probability , If it is less than the significance level 0.05, The probability that the two are irrelevant is very small , To exclude .

Case microfinance chi square test results ：

 

1234 It's more important ,5678 Is not important

T Inspection process ： to F test , How to T test

 

 

lower than 0.05 As an important variable

ANOVA Inspection process ： Find out first F-value, How to find T-value

 

The result is very close to .

Pearson correlation coefficient ：

Selection of highly relevant features （ Redundant variables ）：

Highly relevant fields often appear , The information is repeated , Using Pearson correlation coefficient , Check the correlation between the two . Greater than 0.95 Just erase the variables .

It depends on keeping that , Variable can be found 1 And variables 2 Relationship with goals .

1. 1. 1. Feature selection of model mode

Decision tree 、 Logical regression , Random forests ,XGBoost

The model will automatically select the most important variables , Variables that do not have collinearity ,

It can solve collinearity , Irrelevant issues .

1. 1. 1. RFECV（ Recursive variable selection .）

Cross validation method to verify .CV.

RFE： repeat

The evaluation index can use the index you decide . Remove Variable , If the index gets worse

 

 

backward ： First use cross validation , Get the index value , Remove one of them , After the indicators get better , Continue to remove , If the index value becomes worse , Just go back and don't eliminate .

3 Methods , Forward method , backward , Stepwise regression

The best effect , But it consumes more energy , A waste of time