Introduction to anomaly detection

One 、 Understand the basic concepts of anomaly detection
1、 What is anomaly detection
Anomaly detection refers to identifying data that is significantly different from the distribution of the subject distribution data or from the predicted distribution data .
2、 Types of exceptions
2.1 Point anomaly
It means that a few individuals are abnormal , Most individuals are normal .
2.2 Context exception
It refers to the specific environment , Individual instances are normal , But it is abnormal in other environments . for example ： The temperature suddenly rises or falls at a certain time .
2.3 The group is abnormal
It refers to the occurrence of abnormal individuals in a group , But this individual instance is not really abnormal , What is really unusual is this group . for example ： An abnormal subset of fake accounts in Social Networks , But the individuals in this subset are as normal as the real accounts .
3、 Anomaly detection task classification
Supervised
Unsupervised
Semi supervision
Two 、 When to use the exception detection method
1、 Processing of characteristic data ： When doing Feature Engineering , In order to prevent abnormal data from affecting normalization , You need to filter out abnormal data ; When doing the second classification , There is very little training sample data , The sample category data is unbalanced , Use unsupervised outlier detection .
2、 Log exception detection 、 Network intrusion anomaly detection 、 Credit card anomaly detection, etc .

3、 ... and 、 Method of anomaly detection
1、 traditional method
1.1 Methods based on statistics
Assume that normal data objects are given by a unified model , What does not conform to the model is abnormal data . The statistical method of anomaly detection is ： Learn a generation model that fits a given data set , Then identify the objects in the low probability region of the model , Take them as outliers .
1.2 Linear model
PCA
2、 Integration method
3、 Open source libraries are often used for exception detection pyod