Nonlinear optimization: steepest descent method, Newton method, Gauss Newton method, Levenberg Marquardt method

Nonlinear optimization

For a minimum nonlinear multiplication problem ：
$$\underset{x}{\min }F(x)=\frac{1}{2}{\Vert \begin{array}{c}f(x)\end{array}\Vert }_2^2$$
Find the function in the formula $f(x)$ Of L2 The minimum value of half of the sum of squares of norms ,L2 Norm refers to the square operation after taking the sum of squares of each item .

Solve the minimum objective function , Into the independent variable when its derivative is zero $x$ Value ：
$$\frac{dF}{dx}=0$$
It is generally used iteration To solve , Start with an initial value , Update variables iteratively , Make the value of the objective function drop ：
$$\begin{array}{rl}1.& tosetfirstbeginningvalue{x}_0\\ 2.& YesOnThe\; firstkTimeOverlappinggeneration,Look\; forlook\; forincreaseThe\; amount\Delta x_{k}sendhave\; to{\Vert \begin{array}{c}f(x_k+\Delta x_k)\end{array}\Vert }_2^{2}reachToextremelySmallvalue\\ 3.& if\Delta x_{k}footenoughSmall,befullfootstripPieces\; ofstopstop\\ 4.& backAnd,x_{k+1}=x_k+\Delta x_k,FollowingTo\; continueOverlappinggenerationNextOneround\end{array}$$
Solve increment $\Delta x_k$ The way , There are usually the following ： The steepest descent 、 Newton method 、 gaussian - Newton's method and lewenberg - Marquardt method et al .

The steepest descent

Consider the $k$ Sub iteration , Make the objective function First order Taylor expansion ：
$$F(x_k+\Delta x_k)\approx F(x_k)+J(x_k{)}^T\Delta x_k$$
matrix $J(x_k)$ For the function $F(x)$ About $x$ First derivative of , be called Jacobian matrix （Jacobian Matrix）.

At this time, you can select the increment as follows ：
$$\Delta x^{\ast }=arg\min (F(x_k)+J(x_k{)}^T\Delta x_k)$$
Right side to $\Delta x_k $ Derivation and zero ：
$$\Delta x^{\ast }=-J(x_k)$$
Usually , A step size needs to be further defined $\lambda$, So as to advance along the direction gradient , Make the objective function descend under the first-order linear approximation . The steepest descent method is too greedy , As a result, its descent route is easy to form a sawtooth route （ Go to the edge of the grid ）, Thus increasing the number of iterations .

Newton method

Consider the $k$ Sub iteration , Make the objective function Second order Taylor expansion ：
$$F(x_k+\Delta x_k)\approx F(x_k)+J(x_k{)}^T\Delta x_k+\frac{1}{2}\Delta x_k^{T}H(x_k)\Delta x_k$$
matrix $H(x_k)$ For the function $F(x)$ About $x$ The second derivative of , be called Hazen matrix （Hessian Matrix）, At this time, the increment is as follows ：
$$\Delta x^{\ast }=arg\min (F(x_k)+J(x_k{)}^T\Delta x_k+\frac{1}{2}\Delta x_k^{T}H(x_k)\Delta x_k)$$
Again , The right derivative takes zero ：
$$H(x_k)\Delta x_k=-J(x_k)$$
Solve the above linear equation , You can get the incremental expression . Newton's method requires a lot of computational effort to solve $H$, This problem should be avoided .

The quasi Newton method is usually used to solve the least square problem .

gaussian - Newton method

For an objective function $F(x)$：
$$\underset{x}{\min }F(x)=\frac{1}{2}{\Vert \begin{array}{c}f(x)\end{array}\Vert }_2^2$$
Gauss Newton method passes $f(x)$ Do Taylor expansion , Instead of $F(x)$ Target expansion , To improve efficiency .
$$f(x_k+\Delta x_k)\approx f(x_k)+J(x_k{)}^T\Delta x_k$$
here , Solve the increment so that ${\Vert \begin{array}{c}f(x+\Delta x_k)\end{array}\Vert }^2$ Minimum ：
$$\Delta x^{\ast }=arg\underset{x}{\min }\frac{1}{2}{\Vert \begin{array}{c}f(x_k)+J(x_k{)}^T\Delta x_k\end{array}\Vert }^2$$
Simplify the expression ：
$$\begin{array}{rl}\frac{1}{2}{\Vert \begin{array}{c}f(x_k)+J(x_k{)}^T\Delta x_k\end{array}\Vert }^2& =\frac{1}{2}(f(x_k)+J(x_k{)}^T\Delta x_k{)}^T(f(x_k)+J(x_k{)}^T\Delta x_k)\\ & =\frac{1}{2}({\Vert \begin{array}{c}f(x_k)\end{array}\Vert }_2^2+2f(x_k)J(x{)}^T\Delta x_k+\Delta x_k^{T}J(x_k)J(x_k{)}^T\Delta x_k)\end{array}$$
Solve to minimize $x$ That's right $\Delta x$ The derivative is zero ：
$$\begin{array}{r}J(x_k)f(x_k)+J(x_k)J^T(x_k)\Delta x_k=0\\ J(x_k)J^T(x_k)\Delta x_k=-J(x_k)f(x_k)\end{array}$$
remember $H(x_k)=J(x_k)J^T(x_k)$,$g(x_k)=-J(x_k)f(x_k)$, So as to get information about $\Delta x_k$ A linear system of equations ：
$$H(x_k)\Delta x_k=g(x_k)$$
Call it The incremental equation , or Gauss Newton equation （Gauss-Newton Equation）, Normal equation （Normal Equation）. gaussian - Newton's method uses $J{J}^T$ Second order Hessian matrix in approximate Newton method $H$, So as to avoid a lot of calculations , The algorithm flow is as follows ：
$$\begin{array}{rl}1.& tosetfirstbeginningvalue{x}_0\\ 2.& YesThe\; firstkTimeOverlappinggeneration,seekJas.canThanMomentfrontJ(x_k)andBy\; mistakeBadf(x_k)\\ 3.& seekExplainincreaseThe\; amountFangcheng：H\Delta x_k=g\\ 4.& if\Delta x_{k}footenoughSmall,befullfootstripPieces\; ofstopstop\\ 5.& backAnd,x_{k+1}=x_k+\Delta x_k,FollowingTo\; continueOverlappinggenerationNextOneround\end{array}$$

For the solution of incremental equation , The matrix should be satisfied $H$ reversible , but $J{J}^T$ Is a positive semidefinite matrix , Singular matrices or ill conditioned situations may occur , This leads to the non convergence of the algorithm . Levenberg - Marquardt method corrects the above problem to a certain extent .

Levenberg - The Marquardt method

Levenberg - The convergence speed of Marquardt method is slower than Gauss - Newton method , But it is more robust , Also known as damped newton method （Damped Newton Method）

gaussian - Newton's second-order Taylor expansion to approximate linearization , Its effect only has a good approximate effect near the expansion point . So deal with $\Delta x_k$ Add an interval range , be called Confidence interval （Trust Region）. It is considered that it is approximately valid within the confidence interval and invalid outside the approximate interval .

The difference between the approximate model and the actual function is used to determine the scope of the confidence interval ：
$$\rho =\frac{f(x_k+\Delta x_k)-f(x_k)}{J(x_k{)}^T\Delta x}$$
indicators $\rho$ Used to describe the good or bad degree of approximation , among , The denominator is the value decreased by the approximate model , The numerator is the decreasing value of the actual function .

$\rho$ near 1, The approximate effect is good , The approximate range should be expanded ;$\rho$ smaller , The approximation effect is poor , The approximate range should be reduced . thus , Build Levenberg - Marquardt algorithm model ：
$$\begin{array}{rl}1.& tosetfirstbeginningvalue{x}_{0}WithAndfirstbeginningoptimalturnAnd\; a\; halfpath\mu \\ 2.& YesThe\; firstkTimeOverlappinggeneration,stayhighSicattleTonLawOfThe\; baseFoundationOnincreaseAddLetterlaiDistrictDomain：\\ & \underset{\Delta x_k}{\min }\frac{1}{2}{\Vert \begin{array}{c}f(x_k)+J(x_k{)}^T\Delta x_k\end{array}\Vert }^2,s.t.{\Vert \begin{array}{c}D\Delta x_k\end{array}\Vert }^2\le \mu ,\\ & Itsin,\mu byLetterlaiDistrictbetweenAnd\; a\; halfpath,DbysystemCountMomentfront\\ 3.& metercountfingermark：\rho =\frac{f(x_k+\Delta x_k)-f(x_k)}{J(x_k{)}^T\Delta x}\\ 4.& if\rho >0.75,beset\; upSet\; up\mu =2\mu \\ 5.& if\rho <0.25,beset\; upSet\; up\mu =0.5\mu \\ 6.& if\rho It\text{'}s\; aboutOnsomethresholdvalueDistrictbetween,berecognizebynearlikecanThat\text{'}s\; ok,x_{k+1}=x_k+\Delta x_k\\ 7.& sentencebreakyesnoclosedConvergence,ifclosedConvergencebejunctionbeam,backAndreturnreturnThe\; firstTwoStepOverlappinggeneration\end{array}$$
here , Limit the increment value to the radius $\mu$ In the ball （${\Vert \begin{array}{c}\Delta x_k\end{array}\Vert }^2\le \mu$）, After adding the coefficient matrix , It can be regarded as an ellipsoid （${\Vert \begin{array}{c}D\Delta x_k\end{array}\Vert }^2\le \mu$）.

Levenberg takes $D=I$, That is, incremental constraint in the ball . And Marquardt will $D$ Take it as a non negative diagonal matrix （ In practice, we usually take $J^{T}J$ Square root of diagonal element ）, Thus, the constraint range on the dimension with small gradient is larger .

Construct Lagrange equation , Put the constraint into the objective function ：
$$L(\Delta x_k,\lambda )=\frac{1}{2}{\Vert \begin{array}{c}f(x_k)+J(x_k{)}^T\Delta x_k\end{array}\Vert }^2+\frac{\lambda }{2}({\Vert \begin{array}{c}D\Delta x_k\end{array}\Vert }^2-\mu )$$
call $\lambda$ Is the Lagrange multiplier , Ask about $\Delta x_k$ The derivative of is zero ：
$$(H+\lambda D^{T}D)\Delta x_k=g$$
When parameters $\lambda$ More hours ,$H$ Occupy a dominant position , The quadratic approximation model is better in the range , Levenberg - Marquardt method is close to Gauss Newton method ; When parameters $\lambda$ large ,$\lambda D^{T}D$ Occupy a dominant position , The quadratic approximation model is poor in the range , Levenberg - The Marquardt method is close to the steepest descent method . Levenberg - Marquardt method to a certain extent , The coefficient matrix of linear equations can be avoided to be nonsingular 、 Sick questions , Provide more stability 、 More accurate increments $\Delta x_k$