Zebras are recognized as dogs, and the reason for AI's mistakes is found by Stanford

Pine From the Aofei temple
qubits | official account QbitAI

It's obviously a zebra ,AI Why is it a dog ？

Classification models sometimes misjudge categories when classifying images .

Learned AI, And screw up some predictions , There must be a mistake in one of the links .

Two doctoral students and professors from Stanford University James Zou In a paper , Take us to explore the reasons why the classification model makes mistakes .

And then , This paper proposes a method —— Conceptual interpretation of counterfactual （Conceptual Counterfactual Explanations）, And evaluated its effect .

In this way , We can redefine the classification criteria of the model , So as to explain AI The reason for the mistake .

Let's see .

AI Why make mistakes ？

Want to improve the accuracy of the subsequent prediction of the classification model , First we have to figure out what went wrong with this model .

Generally speaking ,AI The following reasons are responsible for botched predictions ：

First It is in the process of actual prediction of the model , The classification criteria may deviate from the pre training , This makes the training model ineffective in the actual prediction process , And then reduce the accuracy of the prediction results .

for instance , Pathologists use pre trained models to classify histopathological images , But the effect on his image is not very good .

This may be in his image , The tone is different from the original training data .

secondly , In the process of model learning , You may learn something called “ Pseudo correlation ” Things that are , That is, some seemingly unrelated elements are associated with the recognized image .

Let's look at this example ：

In this model training process , There is snow in the photos of all dogs in the sample , This leads the model to associate snow with dogs , And mispredict ： A dog without snow is not a dog .

This may be the data set used , Are collected in the same scene , It will hinder the generalization of the model .

besides , It may also be when training the model , Some artificial deviations .

for example , A dermatologist uses trained AI To classify skin diseases in the image , But the effect of other colleagues is not satisfactory .

This may be because in the training sample , The complexion of the skin is single 、 And the age distribution is narrow .

I understand AI“ Make a mistake ” After the reason for , How can we accurately judge what is wrong with the model ？

AI Make a mistake , It explains

James Zou In this paper, a new method called Conceptual interpretation of counterfactual （CCE） Methods .

say concretely , In this way , To explore input data And the predicted results , Finally find the error of the model .

that CCE How to explain it ？

Define concept base

The first thing to do , It is to set up and refine a concept base C, That is to make a classification standard .

say concretely , Concept Library C The concepts in can be used to classify images , Such as device c1（ The street 、 Snow, etc. ）、 Picture quality c2（ Clear 、 Fuzzy etc ）······

such , You can get a set of interpretable concept libraries C={c1,c2,…}.

then , You need to find corresponding training data for each of these concepts .

Concrete , Is to collect the data that is consistent with it （Pci） Is inconsistent with （Nci） Example , Generally speaking, the quantity should be the same （Pci=Nci=100）.

For each concept ,CCE We should learn their classification methods and “ Way of thinking ”.

Through two methods ：

One is through learning support vector machines （SVM）, To find an algorithm that can distinguish the best way of two things （ Linear classifier ）.

The other is to learn the corresponding concept activation vector （CAV）, It can be used to explain the specific reasons why images are incorrectly classified .

As the figure below , They are all images of zebras , The reasons for the wrong classification are different .

This step only needs to be done once for each model you want to evaluate , after CAV It can be used to explain any number of misclassification .

Given error classification criteria

We can change the proportion of different concepts in the model , Adjust the classification standard accordingly , These adjustments should meet the following principles ：

1、 correctness ： If a classification standard achieves the expected results , Then it is considered to be correct .

2、 effectiveness ： Classification standards should not violate the basic human cognition .

3、 sparsity ： The ultimate goal is to communicate the mistakes of the model to users , Too many variables are not conducive to effectively communicating information .

Our goal is to make the prediction results as close to the training results as possible , That is, minimize cross entropy loss .

Therefore, it is necessary to constantly optimize the standard of model prediction , By adjusting the standards to be modified , Weight it , Finally, the effect of correcting the wrong classification is achieved .

After understanding , Let's take a look through a practical example , How to use it? CCE“ Probe ” Where the classification model goes wrong .

ad locum , The classification model incorrectly recognizes zebra images as African hounds .

therefore , We first generate this model to recognize zebras as dogs by a series of Standards .

then , Rate these criteria , If the score is positive , It means adding this concept to the image , It will improve the probability of correct classification , vice versa .

In this case , If you add stripes（ stripe ） The concept , The probability of recognizing it as a zebra will be higher .

stay c） In the figure , adopt CCE Analysis can also be seen intuitively ,“Polka Dots”（ speckle ） and “Dog”（ Dog ） It is the reason for the wrong prediction of the model .

CCE How does it work ？

See here , You must be right CCE Have a preliminary understanding of the principle of .

Then does it judge accurately , What's the effect ？

CCE Purpose , It mainly reveals what the model learned in the training process “ Pseudo correlation ”, With it, you can capture other things in the image “ Irrelevant elements ” False correlation with image .

Tests found , in the majority of cases , The model is in exceed 90% False correlation is identified in the test samples of the wrong classification .

Look at this form , Compared to other methods , Use CCE, The probability of identifying pseudo correlation in the sample is the highest .

CCE can accurate Identify the pseudo correlation in the sample , Let's look at this example ：

Change the color of the apple picture （ Make the picture gray ）, When the probability of recognition error of classification model increases （ Black line ）,CCE Identify “ green ” The higher the score for pseudo Correlation （ The green line ）.

besides ,CCE It also has potential in the field of Medicine .

image Abubakar Abid Wait for someone to use CCE, In dermatology （ Skin condition classification ）、 Cardiology in chest X-ray images （ Pneumothorax classification ） Have done relevant tests .

CCE Use the learned deviation and image quality conditions to explain the model errors , It has also been confirmed by professional dermatologists —— These factors , Indeed, it is largely the reason why skin images are difficult to classify .

Besides ,CCE The speed is also very fast .

The concept base only needs to be learned once using a simple support vector machine , Each test example is in a single CPU It takes less time than 0.3s.

It is important to , It can be easily applied to any deep network , Detect the cause of model error without training data .

If you are interested in this method , If you want to try it yourself , You can click the following link to view it .

The authors introduce

James Zou , Paper correspondent , He is an assistant professor in the Department of biomedical data science at Stanford University , Assistant professor of computer science and Electrical Engineering .

On 2014 Received a doctorate from Harvard University in , He was a member of Microsoft Research 、 Gates scholar at Cambridge University and Simmons researcher at the University of California, Berkeley .

His research has been Sloan Fellowship、NSF CAREER Award as well as Google、Amazon And Tencent AI The support of awards .

Abubakar Abid （ front ）、 Mert Yuksekgonul（ after ） First author of the paper , All are doctoral students of Stanford University .

Reference link ：

1、https://arxiv.org/pdf/2106.12723.pdf
2、https://github.com/mertyg/debug-mistakes-cce
3、https://twitter.com/james_y_zou/status/1541452062344417280