Performance measure of classification model

Confusion matrix （Confusion matrix）

For dichotomies , The actual data can be divided into positive examples and negative examples . According to the discrimination category of the model and the actual category of the data , Four results can be obtained ： Real examples （True positive）, False positive example （False positive）, True counter example (True negative), False counter example (False negative).

If the discrimination category is consistent with the actual category , It is true , atypism , False , in other words , Real examples Means , The classification of the model is consistent with the actual , All positive examples ; and False positive example The meaning of is false positive example ： The discrimination is positive , But it's wrong （ Actually, it is a counterexample ）; False counter example It's a false counterexample ： It is judged as a counterexample , But the judgment is wrong , In fact, it is a positive example .

use TP、FP、TN、FN To represent the number of corresponding results , The confusion matrix of classification results can be obtained ：

-	The prediction is positive	Negative prediction
Actual positive example	TP	FN
Actual counterexample	FP	TN

Consider an example , Now there is 100 people , One of the boys 70 people , girl student 30 people . There is a model to classify boys and girls . The discrimination result of the model is ： schoolboy 60 people （ What is really a boy is 55 people , rest 5 People are girls ）, girl student 40 people （ What is really a girl is 25 people , in addition 15 For boys ）. Then the confusion matrix can be expressed as ：

-	Predicted to be male	Predicted to be female
Actual boys	55	15
Actual girls	5	25

Commonly used evaluation index

Accuracy

The accuracy of Chinese Translation / precision .
It refers to , The proportion of samples with correct classification in the total sample books .
In the confusion matrix ,TP and TN All belong to the samples with correct classification , therefore ,
$$Accuracy=\frac{TP+TN}{TP+FP+FN+TN}$$

Use the example of male and female students above to calculate Accuracy Words ,acc=(55+25)/100=0.80

Precision

Chinese translation is accuracy / Precision rate .
It refers to the sample in which the model is judged as a positive example , How many are real positive examples , therefore ,
$$Precision=\frac{TP}{TP+FP}$$
Common scenarios are “ How much information is retrieved that users are really interested in ”.
Use the example of male and female students above to calculate Precision Words ,precision=55/(55+5)=0.917

Recall

Recall rate / Recall rate .
It refers to all positive examples , How many are judged as positive examples by the model , therefore
$$Recall=\frac{TP}{TP+FN}$$
Common scenarios are “ In the information retrieval of all fugitives , How many fugitives can be detected ”.
Use the example of male and female students above to calculate Recall Words ,recall=55/(55+15)=0.786

F1 value

frequently-used F1 The value is calculated as ：
$$F1=\frac{2\ast Precision\ast Recall}{Precision+Recall}=\frac{2\ast TP}{sampleBentotalCount+TP-TN}$$
Use the example of boys and girls above to calculate ,F1=（20.9170.786）/（0.917+0.786）=0.846, Or is it F1=（2*55）/（100+55-25）=0.846

PR curve

PR The curve is based on precision For the vertical axis ,recall Is the horizontal axis , The curve drawn .
PR The larger the area under the curve , The better the performance .
On the curve , When precision And recall When the values of are equal , This point is the equilibrium point (Break-Even Point).

ROC curve

ROC Its full name is “ Work characteristics of subjects ”.
The vertical axis is the true case rate （TPR）, The horizontal axis is false positive rate （FPR）.
$$TPR=\frac{TP}{TP+FN}$$
$$FPR=\frac{FP}{TN+FP}$$

AUC

Express Area under ROC Cureve, yes ROC The area under the curve .
Larger area , The better the performance .

Cost curve The cost curve

The previous evaluation criteria focus on TP, Actually FP and FN It is also important in some scenarios .
For example, a medical scene ： The cancer diagnosis classifier classifies healthy patients into cancer patients , Or classify cancer patients as healthy patients . These two scenarios are the misclassification of the model , But the consequences of the latter situation are more serious .
therefore , In order to better measure the different losses caused by different mistakes , Give fault to “ Unequal costs ”（unequal cost）.
At the time of calculation , We need to optimize the overall cost （total cost）.