Gaussian distribution and its maximum likelihood estimation

高斯分布

一维高斯分布

The probability density function of a one-dimensional Gaussian distribution is ：
$$N(\mu ,{\sigma }^2)=\frac{1}{\sqrt{2\pi }\sigma }\exp (-\frac{(x-\mu {)}^2}{2{\sigma }^2})$$
多维高斯分布

$D$ The probability density function of the dimensional Gaussian distribution is ：
$$N(\mu ,\Sigma )=\frac{1}{(2{\pi }^{\frac{D}{2}}|\Sigma {|}^{\frac{1}{2}})}\exp (-\frac{(x-\mu {)}^2{\Sigma }^{-1}(x-\mu )}{2})$$

极大似然估计

贝叶斯公式

贝叶斯公式如下：
$$P(\theta |X)=\frac{P(X|\theta )P(\theta )}{P(X)}$$
其中,$P(X|\theta )$ 称为后验概率,$P(\theta )$ 称为先验概率,$P(\theta |X)$ be the likelihood function.The so-called maximum likelihood estimation,Even if you want to let the likelihood function $P(\theta |X)$ 取到最大,Estimate the parameters at this time $\theta$ 的值.详见：先验、后验、似然.

高斯分布的极大似然估计

假设我们有 $N$ 个观测数据 $X=(x_1,x_2,\dots ,x_N)$ ,Each sample point is $D$ 维的,Then our data is one $N\times D$ 的矩阵.The parameter we want to estimate is the mean in the multidimensional Gaussian distribution $\mu$ 和协方差矩阵 $\Sigma$ .

Here we take the one-dimensional Gaussian distribution as an example for derivation.i.e. each sample point $x_i$ 是一维的,What we want to estimate is the mean of a one-dimensional Gaussian distribution $\mu$ 和方差 ${\sigma }^2$ ,即 $\theta =(\mu ,{\sigma }^2)$ .

Below we use maximum likelihood estimation to estimate these two parameters：
$${\hat{\theta }}_{MLE}=\arg \underset{\theta }{\max }\mathcal{L}(\theta )$$
为了方便计算,We usually optimize the log-likelihood,有：
$$\begin{array}{rl}\mathcal{L}(\theta )& =\log P(X|\theta )\\ & =\log \prod _{i=1}^{N}P(x_i|\theta )\\ & =\sum _{i=1}^N\log P(x_i|\theta )\\ & =\sum _{i=1}^N\log \frac{1}{\sqrt{2\pi }\sigma }\exp (\frac{(x_i-\mu {)}^2}{2{\sigma }^2})\\ & =\sum _{i=1}^N[\log \frac{1}{\sqrt{2\pi }}+\log \frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}]\end{array}$$
And can throw away the constant term in it,is the final optimization goal：
$${\hat{\theta }}_{MLE}=\arg \underset{\theta }{\max }\sum _{i=1}^N[\log \frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}]$$
接下来我们分别对 $\mu$ 和 ${\sigma }^2$ 求偏导,并令其等于零,得到估计值.

对于 $\mu$ ：
$$\begin{array}{rl}{\hat{\mu }}_{MLE}& =\arg \underset{\mu }{\max }\sum _{i=1}^N[-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}]\\ & =\arg \underset{\mu }{\min }\sum _{i=1}^N(x_i-\mu {)}^2\end{array}$$
求偏导：
$$\frac{\partial {\sum }_{i=1}^N(x_i-\mu {)}^2}{\partial \mu }=\sum _{i=1}^N-2\times (x_i-\mu )\triangleq 0$$
得到：
$${\hat{\mu }}_{MLE}=\frac{1}{N}\sum _{i=1}^N{x}_i$$
对于 ${\sigma }^2$
$$\widehat{{\sigma }^2}=\arg \underset{{\sigma }^2}{\max }\sum _{i=1}^N[\log \frac{1}{\sigma }-\frac{(x_i-\mu {)}^2}{2{\sigma }^2}]=\arg \underset{{\sigma }^2}{\max }{\mathcal{L}}_{{\sigma }^2}$$
求偏导：
$$\begin{gathered} \frac{\partial {\mathcal{L}}_{{\sigma }^2}}{\partial \sigma }=\sum _{i=1}^N[-\frac{1}{\sigma }-\frac{1}{2}(x_i-\mu )\times (-2)]\triangleq 0 \\ \sum _{i=1}^N[-{\sigma }^2+(x_i-\mu {)}^2]\triangleq 0 \\ \sum _{i=1}^N{\sigma }^2=\sum _{i=1}^N(x_i-\mu {)}^2 \end{gathered}$$
得到：
$${\widehat{{\sigma }^2}}_{MLE}=\frac{1}{N}\sum _{i=1}^N(x_i-{\hat{\mu }}_{MLE}{)}^2$$

有偏估计和无偏估计

有偏估计（biased estimate）是指由样本值求得的估计值与待估参数的真值之间有系统误差,其期望值不是待估参数的真值.

在统计学中,估计量的偏差（或偏差函数）是此估计量的期望值与估计参数的真值之差.偏差为零的估计量或决策规则称为无偏的.否则该估计量是有偏的.在统计学中,“偏差”是一个函数的客观陈述.

我们分别计算 ${\hat{\mu }}_{MLE}$ 和 ${\widehat{{\sigma }^2}}_{MLE}$ ,Let us examine whether the two estimates are unbiased.

对于 ${\hat{\mu }}_{MLE}$
$$E[{\hat{\mu }}_{MLE}]=E[\frac{1}{N}\sum _{i=1}^N{x}_i]=\frac{1}{N}\sum _{i=1}^{N}E{x}_i=\mu$$
可以看到,${\hat{\mu }}_{MLE}$ The expectation is equal to the truth value $\mu$ ,So it is an unbiased estimate.

对于 ${\widehat{{\sigma }^2}}_{MLE}$
$$\begin{array}{rl}{\widehat{{\sigma }^2}}_{MLE}& =\frac{1}{N}\sum _{i=1}^N(x_i-{\hat{\mu }}_{MLE}{)}^2\\ & =\frac{1}{N}\sum _{i=1}^N(x_i^2-2\times x_i\times {\hat{\mu }}_{MLE}+{\hat{\mu }}_{MLE}^2)\\ & =\frac{1}{N}\sum _{i=1}^N(x_i^2-2{\hat{\mu }}_{MLE}^2+{\hat{\mu }}_{MLE}^2)\\ & =\frac{1}{N}\sum _{i=1}^N(x_i^2-{\hat{\mu }}_{MLE}^2)\end{array}$$
求期望：
$$\begin{array}{rl}E[{\widehat{{\sigma }^2}}_{MLE}]& =E[\frac{1}{N}\sum _{i=1}^N(x_i^2-{\hat{\mu }}_{MLE}^2)]\\ & =E[\frac{1}{N}\sum _{i=1}^N((x_i^2-{\mu }^2)-({\hat{\mu }}_{MLE}^2-{\mu }^2))]\\ & =E[\frac{1}{N}\sum _{i=1}^N(x_i^2-{\mu }^2)]-E[\frac{1}{N}\sum _{i=1}^N({\hat{\mu }}_{MLE}^2-{\mu }^2)]\\ & =\frac{1}{N}\sum _{i=1}^{N}E(x_i^2-{\mu }^2)-\frac{1}{N}\sum _{i=1}^{N}E({\hat{\mu }}_{MLE}^2-{\mu }^2)\\ & =\frac{1}{N}\sum _{i=1}^N[E(x_i^2)-E({\mu }^2)]-\frac{1}{N}\sum _{i=1}^{N}E({\hat{\mu }}_{MLE}^2)-E({\mu }^2)\\ & =\frac{1}{N}\sum _{i=1}^N[E(x_i^2)-{\mu }^2]-\frac{1}{N}\sum _{i=1}^N[E({\hat{\mu }}_{MLE}^2)-{\mu }^2]\\ & =\frac{1}{N}\sum _{i=1}^N[E(x_i^2)-(E{x}_i{)}^2]-\frac{1}{N}\sum _{i=1}^N[E({\hat{\mu }}_{MLE}^2)-E{\hat{\mu }}_{MLE}^2]\\ & =\frac{1}{N}\sum _{i=1}^{N}Var(x_i)-\frac{1}{N}\sum _{i=1}^{N}Var({\hat{\mu }}_{MLE})\\ & =\frac{1}{N}\sum _{i=1}^N{\sigma }^2-\frac{1}{N}\sum _{i=1}^N\frac{{\sigma }^2}{N}\\ & =\frac{N-1}{N}{\sigma }^2\end{array}$$
其中 $Var({\hat{\mu }}_{MLE})=Var(\frac{1}{N}{\sum }_{i=1}^N{x}_i)=\frac{1}{N^2}{\sum }_{i=1}^{N}Var(x_i)=\frac{{\sigma }^2}{N}$.

因此, ${\widehat{{\sigma }^2}}_{MLE}$ is not equal to its true value ${\sigma }^2$ ,And it will be underestimated.An unbiased estimate should be $\frac{1}{N-1}{\sum }_{i=1}^N(x_i-{\hat{\mu }}_{MLE})$ .

Ref

1. 机器学习白板推导
2. 先验、后验、似然
3. 百度百科-有偏估计