Naive Bayes classifier

1、 Classification concept

   Classification is to find out the model that describes and distinguishes data classes or concepts , In order to use the model Object class label with unknown prediction class label .

   Classification is generally divided into two stages ：

• Learning phase ：

  • Create a classifier that describes a predefined data class or concept set .
  • The training set provides the class label of each training tuple , The learning process of classification is also called supervised learning .

• Classification stage ： The process of using a defined classifier for classification .

   Classification and prediction are different concepts , Classification is prediction classification ( discrete 、 disorder ) label , Numerical prediction is to establish a continuous value function model . Classification and category are also different concepts , Classification is supervised learning , Provides the class label of the training tuple ; Clustering is unsupervised learning , Do not rely on training instances with class labels .

2、 naive bayesian classification

2.1 Bayes theorem

   The formula of Bayes theorem is ：
$$P(h│D)=\frac{P(D│h)P(h)}{P(D)}$$
   In style ,D Assume categories for the data to be tested ,$P(h|D)$ yes h Likelihood probability of ,$P(h)$ yes h The prior probability of ,$P(h|D)$ yes h The posterior probability of ,$P(D)$ yes D The prior probability of .

   Let's start with an example ： In a school 60% Of boys (boy),40% The girl of (girl) . Boys always wear trousers (pants), Girls are half in trousers and half in skirts . Randomly select a student wearing trousers , He （ she ） What is the probability of being a girl ？

   The above description can be formalized as ：

   It is known that P(Boy)=60%, P(Girl)=40%, P(Pants|Girl)=50%,P(Pants|Boy)=100% seek ：P(Girl|Pants)

   answer ：
$$P(Girl│Pants)=\frac{P(Girl)P(Pants│Girl)}{P(Boy)P(Pants|Boy)+P(Girl)P(Pants|Girl)}=\frac{P(Girl)P(Pants│Girl)}{P(Pants)}$$

Intuitive understanding ： Figure out how many people wear trousers in school , Then figure out how many girls there are among these people

   For the above problems, we can get such observation knowledge ： In a school 60% Of boys (boy),40% The girl of (girl) . Boys always wear trousers (pants), Girls are half in trousers and half in skirts . Again , We can't directly observe a randomly selected student wearing trousers , Determine whether the student is a boy or a girl .

   For parts that cannot be observed directly , Assumptions are often made . And for the uncertain things , There are often multiple assumptions .

   Bayes provides a way to calculate the posterior probability of hypotheses $P(h|D)$ Methods , That is, the posterior probability is proportional to the product of the prior probability and the likelihood probability .

2.2 Maximum a posteriori hypothesis

   The maximum a posteriori hypothesis learner is in the candidate hypothesis set H Find the given data in D The most likely assumption h,h It is called the maximum a posteriori hypothesis （Maximum a posteriori: MAP）. determine MAP Bayes formula is used to calculate the posterior probability of each candidate hypothesis , The formula is as follows ：

The last step is to remove $P(D)$, Because it is not dependent on h The constant , Or consider that the prior probability of any data is equal .

2.3 Joint probability of multidimensional attributes

   It is known that ： object D Is a vector composed of multiple attributes , Then combined with the above maximum a posteriori Hypothesis , Our goal can be written as ：

   But here comes a problem ： Calculation $P(<a_1,a_2,\dots ,a_n>│h)$ when , When the dimension is too high , Available data becomes sparse , Difficult to get results .

2.4 Independence hypothesis

   The previously mentioned problem of data sparsity can be solved by the assumption of independence , That's assuming D Properties of $a_i$ Independent of each other , Then the above formula can be written as ：

$$\begin{array}{r}P(<a_1,a_2,\dots ,a_n>│h)=\prod _{i}P(a_i|h)\end{array}$$

$$\begin{array}{rl}{h}_{MAP}& =\underset{h\in H}{\max }P(h|<a_1,a_2,\dots ,a_n>)\\ & =\underset{h\in H}{\max }P(<a_1,a_2,\dots ,a_n>│h)P(h)\\ & =\underset{h\in H}{\max }{\coprod }_i^{}P(a_i│h)P(h)\end{array}$$

   After the assumption of Independence , Get an estimate of $P(a_i│h)$ Than $P(<a_1,a_2,\dots ,a_n>│h)$ It's a lot easier . If D The properties of are not mutually independent , The result of naive Bayesian classification is the approximation of Bayesian classification

3、 Bayesian classification case

   The following training set describes the statistics of computer purchase . The characteristics of the training set include age 、 income 、 hobby 、 Credit and purchases .

id	Age	income	hobby	credit	Buy
1	green	high	no	in	no
2	green	high	no	optimal	no
3	in	high	no	in	yes
4	The old	in	no	in	yes
5	The old	low	yes	in	yes
6	The old	low	yes	optimal	no
7	in	low	yes	optimal	yes
8	green	in	no	in	no
9	green	low	yes	in	yes
10	The old	in	yes	in	yes
11	green	in	yes	optimal	yes
12	in	in	no	optimal	yes
13	in	high	yes	in	yes
14	The old	in	no	optimal	no

   Test cases ： A middle-income 、 Young game lovers with good credit , Will you buy a computer ？

   Training set according to the above table , You can get the following training sets of purchased computers . For the following test sets , Judge a middle-income person 、 Whether the young game loving customers with good credit will buy computers .

id	age group	Income situation	hobby	Credit rating	Buy a computer
3	in	high	no	in	yes
4	The old	in	no	in	yes
5	The old	low	yes	in	yes
7	in	low	yes	optimal	yes
9	green	low	yes	in	yes
10	The old	in	yes	in	yes
11	green	in	yes	optimal	yes
12	in	in	no	optimal	yes
13	in	high	yes	in	yes

   First, calculate the probabilities of different attributes among customers who purchase computers in the test set ：
$$\begin{gathered} P(greenyear|purchasebuy)=2/9=0.222 \\ P(closedEnter\; intoinetc.|purchasebuy)=4/9=0.444 \\ P(Lovegood|purchasebuy)=6/9=0.667 \\ P(Letterusein|purchasebuy)=6/9=0.667 \end{gathered}$$
   Then according to the following formula , Calculate the likelihood probability of buying a computer ：

$$P(X|purchasebuy)=0.222\times 0.444\times 0.667\times 0.667=0.044$$
   Again , We can get training sets without buying a computer .

id	age group	Income situation	hobby	Credit rating	Buy a computer
1	green	high	no	in	no
2	green	high	no	optimal	no
6	The old	low	yes	optimal	no
8	green	in	no	in	no
14	The old	in	no	optimal	no

   So the probability of not buying a computer under different attributes in the test set ：
$$\begin{gathered} P(greenyear|Nobuy)=3/5=0.6 \\ P(closedEnter\; intoinetc.|Nobuy)=2/5=0.4 \\ P(Lovegood|Nobuy)=1/5=0.2 \\ P(Letterusein|Nobuy)=2/5=0.4 \end{gathered}$$
   Again , Use the above formula to calculate the likelihood probability of not buying a computer ：

$$P(X|Nobuy)=0.6\times 0.4\times 0.2\times 0.4=0.019$$
   Use formula $P(X|C_i)P(C_i)$, Available ：
$$\begin{gathered} P(C_{buy})=9/14=0.643 \\ P(C_{Nobuy})=5/14=0.357 \\ P(purchasebuy|X)=0.044\times 0.643=0.028 \\ P(Nobuy|X)=0.019\times 0.357=0.007 \end{gathered}$$

4、 How to calculate the probability of continuous data

   The following table describes the results of whether to purchase computers under different income conditions . that , Can the data in the table be used to predict the income of 121, No game hobby 、 Will middle-aged people with good credit buy computers ？

id	income	Buy
1	125	no
2	100	no
3	70	no
4	120	no
5	95	yes
6	60	no
7	220	no
8	85	yes
9	75	no
10	90	yes

   The revenue here is represented by continuous data , Therefore, the previous discrete data probability estimation method cannot be used . For continuous data , We assume that different types of income follow different normal distributions , Using parameters to estimate the expectation and variance of two groups of normal distribution , You can calculate the income as 121 The probability of not buying a computer , As shown below ：

5、 The characteristics of naive Bayesian classifier

• Attributes can be discrete 、 It can also be continuous
• A solid foundation in mathematics 、 Classification efficiency is stable
• Less sensitive to missing and noisy data
• If the attribute is not related , The classification effect is very good

6、 Bayesian algorithm to achieve iris classification

6.1 Introduction to iris

   Iris ( Latin scientific name ：Iris L.), Monocotyledons , Iridaceae perennial herb , The flowers are big and beautiful , High ornamental value . Iris is about 300 Kind of , Iris The dataset contains three of these : Iris iris (Setosa), Variegated Iris (Versicolour), Iris Virginia (Virginica), Each of these 50 Data , Co inclusion 150 Data . Each data contains four attributes : Calyx length , Calyx width , Petal length , Petal width , These four attributes can be used to predict that iris flowers belong to ( Iris iris , Variegated Iris , Iris Virginia ) What kind of .

   Some data in the data set are shown in the figure below ：

6.2 Classification code

import sklearn
#  Import Gaussian naive Bayesian classifier 
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split
import numpy as np
import pandas as pd


data_url = "Iris.csv"
df = pd.read_csv(data_url)
X = df.iloc[:,1:5]
y=df.iloc[:,5]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
#  Use Gaussian naive Bayes to calculate 
######## Begin ########
clf=GaussianNB()
######## End ########
clf.fit(X_train, y_train)
#  assessment 
y_pred = clf.predict(X_test)
acc = np.sum(y_test == y_pred) / X_test.shape[0]
print("Test Acc:%.3f" % acc)