Naive Bayes in machine learning

The training data set is
$T=\{(x_1,y_2),(x_2,y_2),...(x_n,y_n)\}$
Naive Bayes learns joint probability distributions by training data sets $P（X,Y）$.

According to the multiplication formula of probability theory ：$P(X,Y)=P(Y)P(X|Y)$

Learning joint probability can be transformed into learning prior probability distribution and conditional probability distribution .
A priori probability distribution ：
$P(Y=c_k),k=1,2,...,K$

According to the law of large numbers ： When there's enough data , The frequency of direct use can be estimated as probability , So we think that the prior probability can be calculated when the data is sufficient .

Conditional probability distribution ：
$P(X=x|Y=c_k)=P(X^{(1)}=x^{(1)},...X^{(n)}=x^{(n)}|Y=c_k),k=1,1...K$

notes ： The superscript number represents the number in a sample n Attributes or characteristics .

But conditional probability distribution $P(X=x|Y=c_k)$ There are exponentially many parameters , Its estimation is actually infeasible . in fact , hypothesis $x^{(j)}$ There are $S_j$ individual ,j=1,2,…n,Y There are K individual , Then the number of parameters is $K{\prod }_{j=1}^n{S}_j$, in application , This value is much larger than the number of training samples , in other words , Some values do not appear in the training set , It is obviously not feasible to estimate probability by frequency ,“ Not observed ” and “ The probability of occurrence is zero ” It's a different concept .
Naive Bayes assumes conditional independence of conditional probability distribution , This is a strong assumption , Hence the name naive Bayes . The assumption is ：
$P(X=x|Y=c_k)=P(X^{(1)}=x^{(1)},...X^{(n)}=x^{(n)}|Y=c_k)={\prod }_{j=1}^{n}P(X^{(j)}=x^{(j)}|Y=c_k)$
Through this assumption, the parameter is reduced to $K{\sum }_{j=1}^n{S}_j$

Illustrate with examples ： If the value of attribute one is 0,1, The value of attribute 2 is A,B,C, label Y There is only one value of , If we do not use the conditional independence assumption , Then the parameters to be solved are P(0,A),P(0,B),P(0,C),P(1,A),P(1,B),P(1,C)6 individual , The assumption of conditional independence is only P(0),P(1),P(A),P(B),P(C )5 individual , When there are more values for attributes , When there are more attributes , This gap will be even greater .

It is obvious that naive Bayes first modeled joint probability , Then the maximum posterior probability is obtained , It is a typical generative model
A posteriori probability is based on Bayesian theorem ：
$P(Y=c_k|X=c_k)=\frac{P(X=x|Y=c_k)P(Y=c_k)}{{\sum }_1^{k}P(X=x|Y=c_k)P(Y=c_k)}$
Introduce the conditional independence assumption ：
$P(Y=c_k|X=c_k)=\frac{P(Y=c_k){\prod }_1^{j}P(X^{(j)}=x^{(j)}|Y=c_k)}{{\sum }_1^{k}P(Y=c_k){\prod }_1^{j}P(X^{(j)}=x^{(j)}|Y=c_k)}$
Naive Bayesian classifier can be expressed as ：
$y=f(x)=argmax\frac{P(Y=c_k){\prod }_1^{j}P(X^{(j)}=x^{(j)}|Y=c_k)}{{\sum }_1^{k}P(Y=c_k){\prod }_1^{j}P(X^{(j)}=x^{(j)}|Y=c_k)}$
For the denominator, all $c_k$ It's all the same , And we only care about the size relationship and don't care about the specific value , Therefore, the denominator can be omitted and written as ：
$y=f(x)=argmaxP(Y=c_k){\prod }_1^{j}P(X^{(j)}=x^{(j)}|Y=c_k)$

To be updated

To be updated