Least squares system identification class II: recursive least squares

Preface

This paper records the principle of recursive least squares in the course of system identification .

Least squares estimate

The following system identification model ：
$$\begin{gathered} \{\begin{array}{rl}y(1)& ={\phi }^T(1)\theta +v(1)\\ y(2)& ={\phi }^T(2)\theta +v(2)\\ & ...\\ y(t)& ={\phi }^T(t)\theta +v(t)\end{array} \\ \begin{array}{rl}set\; up：Y_t& =[y(1)y(2)...y(t){]}^T\\ H_t& =[\phi (1)\phi (2)...\phi (t){]}^T\\ V_t& =[v(1)v(2)...v(n){]}^T\end{array} \\ Yes：Y_t=H_t\theta +V_t \end{gathered}$$

The least square estimation of parameters can be expressed as ：
$${\hat{\theta }}_{LS}=(H_t^T{H}_t{)}^{-1}{H}_t^T{Y}_t$$

The above method updates the input and output data at each new time , Must be re estimated , The computational load is heavy .

Recursive least squares estimation

principle

Recursive least square estimation uses incremental method , The following is derived ：

$$\begin{gathered} P^{-1}(t)=H_t^T{H}_t=\sum _{i=1}^t\phi (i){\phi }^T(i)=\sum _{i=1}^{t-1}\phi (i){\phi }^T(i)+\phi (t){\phi }^T(t)=P^{-1}(t-1)+\phi (t){\phi }^T(t) \\ \begin{array}{rl}\hat{\theta }(t)& =(H_t^T{H}_t{)}^{-1}{H}_t^T{Y}_t\\ & =P(t)H_t^T{Y}_t\\ & =P(t)\left[\begin{array}{cc}{H}_{t-1}^T& \phi (t)\end{array}\right]{\left[\begin{array}{cc}{Y}_{t-1}^T& y(t)\end{array}\right]}^T\\ & =P(t)H_{t-1}^T{Y}_{t-1}^T+P(t)\phi (t)y(t)\\ & =P(t)P^{-1}(t-1)P(t-1)H_{t-1}^T{Y}_{t-1}^T+P(t)\phi (t)y(t)\\ & =P(t)P^{-1}(t-1)\hat{\theta }(t-1)+P(t)\phi (t)y(t)\\ & =P(t)[P^{-1}(t)-\phi (t){\phi }^T(t)]\hat{\theta }(t-1)+P(t)\phi (t)y(t)\\ & =\hat{\theta }(t-1)-P(t)\phi (t){\phi }^T(t)\hat{\theta }(t-1)+P(t)\phi (t)y(t)\\ & =\hat{\theta }(t-1)+P(t)\phi (t)[y(t)-{\phi }^T(t)\hat{\theta }(t-1)]\end{array} \\ leadThe\; reason\; is：MomentfrontA,B,C,if(I+C{A}^{-1}B)AndAallcanThe\; inverse,be： \\ (A+BC{)}^{-1}=A^{-1}-A^{-1}B(I+C{A}^{-1}B{)}^{-1}C{A}^{-1} \\ {P}^{-1}(t)=P^{-1}(t-1)+\phi (t){\phi }^T(t) \\ holdafterNoodlesOftypeSontakeThe\; inverse,rootAccording\; to\; theleadThe\; reason\; isYes： \\ P(t)=P(t-1)-\frac{P(t-1)\phi (t){\phi }^T(t)P(t-1)}{1+{\phi }^T(t)P(t-1)\phi (t)} \\ P(t)\phi (t)=\frac{P(t-1)\phi (t)}{1+{\phi }^T(t)P(t-1)\phi (t)} \\ L(t)=P(t)\phi (t) \end{gathered}$$

therefore , The recursive least squares of the system parameters can be expressed as ：
$$\begin{gathered} \hat{\theta }(t)=\hat{\theta }(t-1)+L(t)[y(t)-{\phi }^T(t)\hat{\theta }(t-1)]] \\ L(t)=\frac{P(t-1)\phi (t)}{1+{\phi }^T(t)P(t-1)\phi (t)} \\ P(t)=[I-L(t){\phi }^T(t)]P(t-1),P(0)=p_{0}I,p_0=\inf \\ \phi (t)=[...] \end{gathered}$$

New interest , residual

New interest $e(t)=y(t)-{\phi }^T(t)\hat{\theta }(t-1)$
residual $\epsilon (t)=y(t)-{\phi }^T(t)\hat{\theta }(t)$

Data saturation

The phenomenon

When the input-output time series is long enough , The new input and output do not contribute to the result of recursive least squares estimation , It is called data saturation .

reason

$$\begin{gathered} P(t)=[P^{-1}(t-1)+\phi (t){\phi }^T(t){]}^{-1}=>P(t)\le P(t-1) \\ (\alpha I+1/p_0)t\le P^{-1}(t)=P^{-1}(0)+\sum _{i=1}^t\phi (i){\phi }^T(i)\le (\beta I+1/p_0)t \\ \underset{t->\inf }{\lim }{P}^{-1}(t)=0 \end{gathered}$$

resolvent ： Recursive least squares with forgetting factor

$$\begin{gathered} \hat{\theta }(t)=\hat{\theta }(t-1)+L(t)[y(t)-{\phi }^T(t)\hat{\theta }(t-1)]] \\ L(t)=\frac{P(t-1)\phi (t)}{\lambda +{\phi }^T(t)P(t-1)\phi (t)} \\ P(t)=\frac{1}{\lambda }[I-L(t){\phi }^T(t)]P(t-1),P(0)=p_{0}I,p_0=\inf \\ \phi (t)=[...] \end{gathered}$$

Postscript

This paper records the derivation of recursive least squares , Data saturation and its solution . The next chapter will record the least squares ,${\phi }^T(t)$ How to get .