Entropy - conditional entropy - joint entropy - mutual information - cross entropy

0. introduction

It belongs to the basic concept of information theory .

1. Information entropy Entropy (information theory)

• wiki

• How to understand information entropy , This video is really great ！

The amount of information ：$x=lo{g}_{2}N$, $N$ Is the number of possible events . for example , The amount of information for 3, Then the number of original and other possible events is $2^3=8$.

Semaphore is a special case of information entropy ： Events are waiting to happen .

• Suppose a coin ： The probability of positive appearance is 0.8, The probability of the negative side is 0.2
• Turn it into an equi probable event （$N=1/p$）：
  • positive –> Imagine as $1/0.8=1.25$ The probability of one occurrence of an equally probable event
  • The reverse –> Imagine as $1/0.2=5$ The probability of one occurrence of an equally probable event
• Then the amount of information at this time is ： The intuitive amount of information should be $log1.25+log5$, Because the probabilities of these two events are also different , Therefore, the real amount of information at this time is the amount of information after integrating probability ：$0.8\ast log1.25+0.2\ast log5=0.8\ast log\frac{1}{0.8}+0.2\ast log\frac{1}{0.2}$.
• This leads to the famous information entropy formula ：$\Sigma p_{i}log\frac{1}{p_i}=-\Sigma p_{i}log{p}_i$
• $H(X)=-{\sum }_{i=1}^{n}p\left(x_i\right)\log p\left(x_i\right)$

This article The definition of :

• Definition ： entropy , Used to measure the uncertainty of information .

• explain ： The greater the entropy , The more information . The less uncertainty , The smaller the entropy , such as “ Tomorrow the sun will rise in the East ” The entropy of this sentence is 0, Because this sentence does not carry any information , It describes a certain thing .

The example is also very intuitive ：

---

Example ： Suppose there are random variables X, It is used to express the weather of tomorrow .X There are three possible states 1) a sunny day 2) rain 3) overcast The probability of occurrence of each state is P(i) = 1/3, So, according to the formula of entropy ：$H(X)=-{\sum }_{i=1}^{n}p\left(x_i\right)\log p\left(x_i\right)$

It can be calculated that ：H(X) = - 1/3 * log(1/3) - 1/3 * log(1/3) + 1/3 * log(1/3) = log3 =0.47712

If the probability of these three states is (0.1, 0.1, 0.8)：H(X) = -0.1 * log(0.1) *2 - 0.8 * log(0.8) = 0.277528

The previous distribution can be found X The level of uncertainty is very high （ Entropy is very high ）, Every state is very likely . The latter distribution ,X The degree of uncertainty is low （ Low entropy ）, There is a high probability that the third state will occur .

---

2. Conditional entropy Conditional entropy

• wiki

Definition ： Under one condition , The uncertainty of random variables .

• Two random variables X,Y The distribution of , Can form Joint entropy （Joint Entropy）, use H(X, Y) Express . namely ：$H(X,Y)=-\Sigma p(x,y)log(x,y)$
• $H(X|Y)=H(X,Y)-H(Y)$, Express (X, Y) The entropy involved in occurrence , subtract Y The entropy that occurs alone ： stay Y Under the premise of occurrence ,X There's a new entropy .

$$\begin{array}{rl}& H(X|Y)=H(X,Y)-H(X)\\ & =-\sum _{x,y}p(x,y)\log p(x,y)+\sum _{x}p(x)\log p(x)\\ & =-\sum _{x,y}p(x,y)\log p(x,y)+\sum _x\left(\sum _{y}p(x,y)\right)\log p(x)\\ & =-\sum _{x,y}p(x,y)\log p(x,y)+\sum _{x,y}p(x,y)\log p(x)\\ & =-\sum _{x,y}p(x,y)\log \frac{p(x,y)}{p(x)}\\ & =-\sum _{x,y}p(x,y)\log p(y\mid x)\end{array}$$

3. Joint entropy Joint Entropy

• wiki

Two discrete random variables X ,Y The joint entropy of （ In bits ） Defined as :

$$\mathrm{H}(X,Y)=-\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}P(x,y){\log }_2[P(x,y)]$$

For more than two random variables $X_1,...,X_n$ Expand :

$$\mathrm{H}(X_1,...,X_n)=-\sum _{x_1\in {\mathcal{X}}_1}...\sum _{x_n\in {\mathcal{X}}_n}P(x_1,...,x_n){\log }_2[P(x_1,...,x_n)]$$

4. Mutual information Mutual information

• wiki

Definition ： It refers to the degree of correlation between two random variables .

understand ： Determine random variables X After the value of , Another random variable Y The degree to which uncertainty diminishes , Therefore, the minimum value of mutual information is 0, It means that given a random variable has nothing to do with determining another random variable , The maximum value is the entropy of random variable , Means that given a random variable , It can completely eliminate the uncertainty of another random variable . This concept is relative to conditional entropy .

$$\begin{array}{rl}I(X;Y)& \equiv \mathrm{H}(X)-\mathrm{H}(X\mid Y)\\ & \equiv \mathrm{H}(Y)-\mathrm{H}(Y\mid X)\\ & \equiv \mathrm{H}(X)+\mathrm{H}(Y)-\mathrm{H}(X,Y)\\ & \equiv \mathrm{H}(X,Y)-\mathrm{H}(X\mid Y)-\mathrm{H}(Y\mid X)\end{array}$$

$$\begin{array}{rl}I(X;Y)& =\sum _{x\in \mathcal{X},y\in \mathcal{Y}}{p}_{(X,Y)}(x,y)\log \frac{p_{(X,Y)}(x,y)}{p_X(x)p_Y(y)}\\ & =\sum _{x\in \mathcal{X},y\in \mathcal{Y}}{p}_{(X,Y)}(x,y)\log \frac{p_{(X,Y)}(x,y)}{p_X(x)}-\sum _{x\in \mathcal{X},y\in \mathcal{Y}}{p}_{(X,Y)}(x,y)\log p_Y(y)\\ & =\sum _{x\in \mathcal{X},y\in \mathcal{Y}}{p}_X(x)p_{Y\mid X=x}(y)\log p_{Y\mid X=x}(y)-\sum _{x\in \mathcal{X},y\in \mathcal{Y}}{p}_{(X,Y)}(x,y)\log p_Y(y)\\ & =\sum _{x\in \mathcal{X}}{p}_X(x)\left(\sum _{y\in \mathcal{Y}}{p}_{Y\mid X=x}(y)\log p_{Y\mid X=x}(y)\right)-\sum _{y\in \mathcal{Y}}\left(\sum _{x\in \mathcal{X}}{p}_{(X,Y)}(x,y)\right)\log p_Y(y)\\ & =-\sum _{x\in \mathcal{X}}{p}_X(x)\mathrm{H}(Y\mid X=x)-\sum _{y\in \mathcal{Y}}{p}_Y(y)\log p_Y(y)\\ & =-\mathrm{H}(Y\mid X)+\mathrm{H}(Y)\\ & =\mathrm{H}(Y)-\mathrm{H}(Y\mid X).\end{array}$$

---

• Two random variables $X,Y$ Mutual information of , Defined as $X,Y$ The relative entropy of the product of joint distribution and independent distribution of .
• $I(X,Y)=D(P(X,Y)||P(X)P(Y))$ , $I(X,Y)={\sum }_{x,y}p(x,y)\log \frac{p(x,y)}{p(x)p(y)}$

Mutual information and information gain are actually the same value . Information gain = entropy – Conditional entropy , $g(D,A)=H(D)–H(D|A)$

5. Relative entropy

• wiki

Relative entropy , It's also called cross entropy , Cross entropy , Identifying information ,Kullback entropy ,Kullback-Leible Divergence, etc .

set up $p(x)、q(x)$ yes $X$ Two probability distributions of the values in , be $p$ Yes $q$ The relative entropy of is
$$D_{\text{KL}}(P\parallel Q)=\sum _{x\in \mathcal{X}}P(x)\log \left(\frac{P(x)}{Q(x)}\right)=-\sum _{x\in \mathcal{X}}P(x)\log \left(\frac{Q(x)}{P(x)}\right)$$

explain :

• Relative entropy can measure two random variables “ distance ”
• General ,$D(p||q)\ne D(q||p)$