Use 3 in data modeling σ Eliminate outliers for data cleaning

Methods the principle ：

        3σ The criterion is also known as the laida criterion , It assumes that a set of test data contains only random errors , The standard deviation is calculated and processed , Determine an interval according to a certain probability , If the error exceeds this range , It's not a random error, it's a gross error , Data containing this error should be eliminated .
         In the normal distribution σ Represents the standard deviation ,μ For mean .x=μ That is, the axis of symmetry of the image
3σ principle ：
         The values are distributed in （μ-σ,μ+σ) The probability of 0.6827
         The values are distributed in （μ-2σ,μ+2σ) The probability of 0.9544
         The values are distributed in （μ-3σ,μ+3σ) The probability of 0.9974
         It can be said that ,Y The values of are almost all concentrated in （μ-3σ,μ+3σ) Within the interval , There is no possibility of going beyond that 0.3%.

Example data :

	date	Commodity code	sales volumes
0	2020-12-01	A005	1
1	2020-12-01	A014	2
2	2020-12-01	A007	3
3	2020-12-01	A012	4
4	2020-12-01	A009	5
5	2020-12-01	A019	6
6	2020-12-01	A008	7
7	2020-12-01	A019	8
8	2020-12-01	A002	9
9	2020-12-02	A005	10

  Now use 3σ Operation of eliminating outliers .

df_avg = df[' sales volumes '].mean() #  Calculate the mean 
df_std = df[' sales volumes '].std()  #  Calculate the standard deviation 
df['z_score'] = (df[' sales volumes '] - df_avg)/ df_std 
print(display(df))
df = df.loc[(df['z_score']>-3)|(df['z_score']<3)]    #  Filter out data with excessive sales , Eliminate the interference of outliers 
df = df.drop('z_score',axis =1)
print(' mean value ：',df_avg)
print(' Standard deviation ：',df_std)

 mean value ： 5.5
 Standard deviation ： 3.0276503540974917

View the processed results ：

	date	Commodity code	sales volumes
0	2020-12-01	A005	1
1	2020-12-01	A014	2
2	2020-12-01	A007	3
3	2020-12-01	A012	4
4	2020-12-01	A009	5
5	2020-12-01	A019	6
6	2020-12-01	A008	7
7	2020-12-01	A019	8
8	2020-12-01	A002	9
9	2020-12-02	A005	10

         You can see that the result has not changed , Now manually add an outlier , See if it can be filtered out .

        utilize loc Function method to add .

df.loc[10]=['2020-12-02','A007',1000]
df

          Run it again 3σ Operation code for eliminating outliers , View results , The sales quantity of abnormal data is 1000 Has been eliminated

	date	Commodity code	sales volumes
0	2020-12-01 00:00:00	A005	1
1	2020-12-01 00:00:00	A014	2
2	2020-12-01 00:00:00	A007	3
3	2020-12-01 00:00:00	A012	4
4	2020-12-01 00:00:00	A009	5
5	2020-12-01 00:00:00	A019	6
6	2020-12-01 00:00:00	A008	7
7	2020-12-01 00:00:00	A019	8
8	2020-12-01 00:00:00	A002	9
9	2020-12-02 00:00:00	A005	10