Evaluation index of machine learning (II) -- classification evaluation index

See :https://blog.csdn.net/itlilyer/article/details/117880207

Now let's introduce the commonly used evaluation indicators in classification problems .

Before introducing the evaluation indicators, you should first understand " Confusion matrix "." Confusion matrix " Personal understanding is an explanatory matrix that analyzes the different situations of a classification model in predicting the results of an input data .

From the table, we can see that there are four combinations of real values and predicted results ：
real (True Positive)： Put the tag, that is, the true value True The prediction is Positive, such as , Put a picture of a puppy , The label is dog , The prediction result is also for dogs .
True negative (True Negative)： Put the tag, that is, the true value False The prediction is Negative, such as , Put a picture that is not a puppy , Label as other , The prediction results are also for others .
False positive (False Positive)： Put the tag, that is, the true value False The prediction is Positive, such as , Put a picture of kitten , Label as cat , But the prediction is dog .
False negative (False Negative)： Put the tag, that is, the true value True The prediction is Negative, such as , Put a picture of a puppy , The label is dog , The prediction result is not a dog .

The real rate (True Positive Rate,TPR)： Also known as sensitivity , Positive samples predicted to be positive / The total number of actual positive samples ——TPR = $\frac{TP}{TP+FN}$
What a negative rate (True Negative Rate,TNR)： It's also called specificity , Negative samples predicted to be negative / Actual total number of negative samples ——TNR = $\frac{TN}{TN+FP}$
False positive rate (False Positive Rate,FPR)： Negative samples predicted to be positive / Actual total number of negative samples ——FPR = $\frac{FP}{FP+TN}$
False negative rate (False Negative Rate,FNR)： Positive samples predicted to be negative / The total number of actual positive samples ——FNR = $\frac{FN}{TP+FN}$

1. Accuracy rate

Accuracy rate (Accuracy): The number of samples with correct classification / The total number of samples , That is, the prediction result of positive samples is positive , The sum of negative sample prediction results divided by the total number .
ACC = $\frac{TP+TN}{TP+TN+FP+FN}$

2. Average accuracy

Average accuracy (Average per-class Accuracy)： The average of the preparation rates of all categories , This refers to the average value of predicting positive samples as positive and negative samples as negative .
AVE_ACC = $\frac{\frac{TP}{TP+FN}+\frac{TN}{TN+FP}}{2}$

3. Error rate

Error rate ： Number of samples with wrong classification / The total number of samples .
ERROR = $\frac{FP+FN}{TP+TN+FP+FN}$

4. Accuracy

Accuracy (Precision)： It's also called precision rate , See how many of the prediction results are correct . For example, prediction 10 Pictures are dogs , But what is really for dogs is 8 individual , Others are cats and pigs , Then the accuracy is 0.8.
P = $\frac{TP}{TP+FP}$

5. Recall rate

Recall rate (Recall)： It's also called recall , It refers to the proportion of positive samples with correct prediction in all positive samples . Let's say there are 15 Picture of dog , among 12 Zhang predicted a dog , Other predictions are for other animals , Then the recall rate is 0.8.
Recall = $\frac{TP}{TP+FN}$

6.F1

F-Score( Also called F-Measure)： Because under different circumstances, we pay different attention to precision and recall , Some need to reduce prediction errors as much as possible , Some require a higher recall rate .F1 Is the harmonic mean F_β A special case , When β take 1 It degenerates into F1.
F_β = $\frac{(1+{\beta }^2)\ast P\ast R}{({\beta }^2\ast P)+R}$ ; When β=1 when , F1 = $\frac{2\ast P\ast R}{P+R}$

7. ROC and AUC

ROC(Receiver Operating Characteristic), It is often used to evaluate the merits of a binary classifier . A threshold is usually set in logistic regression , If it exceeds the threshold, it is predicted to be positive , Less than the threshold is negative . If the value is reduced, the number of predicted positive classes will increase , At the same time, some samples that are originally negative classes will be identified as positive classes .ROC This phenomenon can be expressed intuitively . We have introduced the real rate above (TPR, True Positive Rate) And the false positive rate (FPR, False Positive Rate),ROC The curve is based on TPR by y Axis ,FPR by x The axis is a curve obtained according to the classification results . If the curve is relatively smooth, there is generally no fitting problem .
We mainly focus on four points and a line in the graph .
The first point **(0, 0)： namely TPR and FPR All are 0, In other words, the classifier predicts all samples as negative classes regardless of positive or negative .
Second points (0, 1)： namely TPR = 1,FPR = 0, That is to say, all samples are classified correctly , Positive samples are predicted to be positive , Negative samples are predicted to be negative .
The third point (1, 0)： namely TPR = 0,FPR = 1, That is to say, all samples are classified incorrectly , The positive sample is predicted to be negative , Negative samples are predicted to be positive .
Fourth point (1, 1)**： namely TPR and FPR All are 1, In other words, the classifier predicts all samples as positive classes regardless of positive or negative .

Through the significance of these points, we can see ROC The closer the curve is to the top left , The better the performance of the classifier .

AUC(Area Under Curve)： yes ROC The area under the curve , The larger the area, the better the classifier . Obviously, the area will not be about 1.

ROC and AUC There is a characteristic ： No matter what the number distribution of positive and negative samples of the tester is ,ROC The curve will not change . In view of the uneven distribution of sample data in the actual data set , Positive samples will be much more or less than negative samples .

8. PR curve

PR The abscissa of the curve is Recall, The ordinate is Precision. One PR The curve should correspond to a threshold （ Statistical probability ）. By selecting the appropriate threshold ( such as 0.5) Divide the samples reasonably , The probability is greater than 0.5 The sample of is a positive example , Less than 0.5 The sample of is negative , After the completion of sample classification, calculate the corresponding accuracy rate and recall rate , Finally, we will get the corresponding relationship , As shown in the figure below .
After many learners learn the data , If one of the learners PR curve A Completely enclose another learner B Of PR curve , It can be asserted that A Better performance than B. however A and B Cross occurs , How to judge the performance ？ We can compare according to the area below the curve , But the more commonly used is the balance point F1. Balance point （BEP） yes P=R The value of time （ The slope is 1）,F1 The bigger the value is. , We can think that the performance of the learner is better .F1 We have already introduced the calculation formula of