Calculus of variations

functional (functional)

• functional $F[y]$ Is a function of a function , That is, its Input is a function $y(x)$, Output is real $F$. This output value depends on one or more functions ( Input ) The integral over a whole path, rather than a general function, depends on discrete variables . For example, calculation The distance between two points , The input is a curve connecting two points , The output is the curve length . stay ML In the scene of , One of the most common functional is entropy $H[x]$, The input of entropy is a random variable $x$ Probability distribution of $p(x)$, Output as real number , So it can also be written $H[p]$

Calculus of Variations

• Variation Is to find a function $y(x)$ bring $F[y]$ Maximum / Minimum . For example, find that the shortest path between two points is a straight line , The probability distribution that maximizes the differential entropy is a normal distribution

---

Euler - The Lagrangian equation

• (1) For the function $y(x)$, Add a small disturbance $ϵ\to 0$ Then Taylor expansion can be used to obtain the following formula ( The first derivative of the polynomial is the same as the original function )
  For multivariable functions $y(x_1,...,x_D)$, Taylor's expansion is as follows ( The first-order partial derivative of the polynomial is the same as the original function )
  Similarly , We can Add a small perturbation to the functional $ϵ\eta (x)$. Usually in the variational method , A functional is an integral , namely
  $$F[y]=\int Gdx$$ Therefore, adding a small disturbance to the functional is equivalent to adding a disturbance to countless variables on the integration path , By extending Taylor expansion of multivariable function, we can get Taylor expansion of functional ：
  among ,$\delta F/\delta y(x)$ Is a functional $F[y]$ Yes $y(x)$ The derivative of
• (2) $G$ It could be a function $y(x)$ and $y(x)$ Functions of derivatives of all orders ( because $y(x)$ yes $x$ Function of , therefore $G$ It's also $x$ Function of ). For the sake of illustration , Let's suppose for the time being $G$ yes $y(x)$ and $y^{\prime }(x)$ Function of , So we can write the functional as ：
  to $F[y]$ A perturbation and Taylor formula expansion can be obtained
  $$\begin{array}{rl}F[y(x)+ϵ\eta (x)]& =\int G(y+ϵ\eta ,y^{\prime }+ϵ{\eta }^{\prime },x)dx\\ & =\int \left[G(y,y^{\prime },x)+ϵ\frac{\partial G}{\partial y}\eta +ϵ\frac{\partial G}{\partial y^{\prime }}{\eta }^{\prime }+O({ϵ}^2)\right]dx\\ & =F[y(x)]+ϵ\int \left[\frac{\partial G}{\partial y}\eta (x)+\frac{\partial G}{\partial y^{\prime }}{\eta }^{\prime }(x)\right]dx+O({ϵ}^2)\end{array}$$ Will be the first 2 Term partial integral can get
  $$\begin{array}{rl}\int \left[\frac{\partial G}{\partial y^{\prime }}{\eta }^{\prime }(x)\right]dx& =\int \frac{\partial G}{\partial y^{\prime }}d\eta (x)\\ & ={\eta (x)\frac{\partial G}{\partial y^{\prime }}|}_x-\int \eta (x)d\frac{\partial G}{\partial y^{\prime }}\\ & =-\int \frac{d}{dx}\left(\frac{\partial G}{\partial y^{\prime }}\right)\eta (x)dx\end{array}$$ The last equation is because $y(x)$ The value of is fixed on the integral boundary , For example, when calculating the distance between two points , The value of the curve at the two endpoints must be the same , So disturbance $\eta (x)$ The value on the integral boundary is 0,$F[y(x)+ϵ\eta (x)]$ You can write the following formula ：
  
• (3) contrast (1) (2) We can know from the formula deduced in
  $$\frac{\delta F}{\delta y(x)}=\frac{\partial G}{\partial y}-\frac{d}{dx}\left(\frac{\partial G}{\partial y^{\prime }}\right)$$ When $y(x)$ Make functional $F[y]$ When taking the extreme value , For all $x$ There must be $\frac{\delta F}{\delta y(x)}=0$, This is because if it exists $\hat{x}$ bring $\frac{\delta F}{\delta y(\hat{x})}\ne 0$, Then you can take a function $\eta (x)$ bring $ϵ\eta (\hat{x})\frac{\delta F}{\delta y(\hat{x})}>0$ And when $x\ne \hat{x}$ From time to tome $\eta (x)=0$. therefore , You can push the following formula , namely Euler - The Lagrangian equation ：
  $$\frac{\partial G}{\partial y}-\frac{d}{dx}\left(\frac{\partial G}{\partial y^{\prime }}\right)=0$$ for example , about
  Euler - The Lagrange equation is
  in addition , If $G$ Only with $y$ of , Ozra - The Lagrange equation is $\frac{\partial G}{\partial y(x)}=0$ ( To any $x$ establish )

---

Example of variational method ： Find the shortest path between two fixed points

• As shown in the above figure, the path is an arbitrary path , Let's take a small segment of micro elements in the area $ds$, It is easy to calculate that the length of the infinitesimal segment is ：
  $$ds\approx \sqrt{(dx{)}^2+(dy{)}^2}=\sqrt{1+y^{\prime 2}}dx$$ The total path length obtained by integration is ：
  $$F[y]={\int }_{x_1}^{x_2}ds={\int }_{x_1}^{x_2}\sqrt{1+y^{\prime 2}}dx$$ The above path length is $y$ Of functional , among $G=\sqrt{1+y^{\prime 2}}$, So there is
  $$\begin{gathered} \frac{\partial G}{\partial y}=0 \\ \frac{d}{dx}\left(\frac{\partial G}{\partial y^{\prime }}\right)=\frac{d}{dx}\left(\frac{y^{\prime }}{\sqrt{1+y^{\prime 2}}}\right)=\frac{\sqrt{1+y^{\prime 2}}{y}^{\prime \prime }-y^{\prime }\frac{y^{\prime }}{\sqrt{1+y^{\prime 2}}}{y}^{\prime \prime }}{1+y^{\prime 2}}=\frac{y^{\prime \prime }}{(1+y^{\prime 2}{)}^{\frac{3}{2}}} \end{gathered}$$ Substitute Euler - Lagrange equation can be obtained
  $$y^{\prime \prime }=0$$ This ordinary differential equation is easy to get $y$ The general solution is $y=c_{1}x+c_2$. This also really shows that the way to minimize the distance between two points on the same plane is a line segment

References

• Bishop, Christopher M., and Nasser M. Nasrabadi. Pattern recognition and machine learning. Vol. 4. No. 4. New York: springer, 2006.
• 【PRML】【 Pattern recognition and machine learning 】【 A formula derived from zero 】 Variation
• Introduction to variational method Part 1.（Calculus of Variations）
• 【 Variational calculation 1】 Euler - The Lagrangian equation