Inferring System and Optimal Control Parameters of Closed-Loop Systems from Partial Observations

TL;DR

The paper tackles inferring both the dynamics $(A,B)$ and the optimal LQR costs $(Q,R)$ from partial, noisy observations of a system operating under closed-loop control. It treats the problem through marginal likelihood of a linear-Gaussian state-space model and leverages the EM algorithm to estimate the closed-loop dynamics, then derives conditions and procedures to disentangle system and cost parameters. A key finding is that infinite-horizon data with only state observations yields non-identifiability of the individual $(A,B,Q,R)$ components, but identifiability can be enhanced by moving to finite horizons or by incorporating partial control observations, with Kleinman-type iterations and Sylvester-DARE relations providing practical recovery procedures. These results offer a principled path to estimate both plant dynamics and control properties in partially observed, closed-loop settings, with potential applications to neuroscience and other domains where control processes are inferred from indirect measurements.

Abstract

We consider the joint problem of system identification and inverse optimal control for discrete-time stochastic Linear Quadratic Regulators. We analyze finite and infinite time horizons in a partially observed setting, where the state is observed noisily. To recover closed-loop system parameters, we develop inference methods based on probabilistic state-space model (SSM) techniques. First, we show that the system parameters exhibit non-identifiability in the infinite-horizon from closed-loop measurements, and we provide exact and numerical methods to disentangle the parameters. Second, to improve parameter identifiability, we show that we can further enhance recovery by either (1) incorporating additional partial measurements of the control signals or (2) moving to the finite-horizon setting. We further illustrate the performance of our methodology through numerical examples.

Figures (4)

• Figure 1: Models of the optimal closed-loop dynamics and observations in the different scenarios analyzed in this work. The infinite-horizon LQR is displayed in (a)-(b). These panels differ in the observations available for inference to an external observer: in (a) the observer only has access to partial and noisy measurements of the state, while in (b) noisy measurements of the control input are also accessible. (c) Finite-horizon LQR and observation model.
• Figure 2: (A) Recovery from observations of the system $\{A,B\}$ and cost function $\{Q,R\}$ parameters in the infinite horizon setting is limited to identification of the closed-loop dynamics $F_{ss}$. We plot the inferred $F_{ss}$ using EM, as well as a combination $\{A,B,Q,R\}$ that yields this same $F_{ss}$. (B) Parameter inference can be reduced to closed-loop identification with probabilistic methods, followed by an investigation of the interaction between the different parameters in setting the closed-loop dynamics.
• Figure 3: Parameter estimation convergence and robustness properties. (A) Matrix 2-norm error between the true parameter $A$ used for simulation ($4\times 4$ unitary rotation) and the estimated $A$ from the iterative procedure in Prop. \ref{['prop:iterative']} as a function of the number of iterations. (B) Similar estimation error to A for $A$ given $Q$ as well as for $Q$ given $A$, this time using $\hat{F}_{ss}$ estimated from the EM-algorithm, as a function of the number of iterations. We use $C=\mathop{\mathrm{I}}\nolimits$ for identifiability. (C-D) Recovery error for $A$ as a function of $Q$ (resp. $F_{ss}$) with perturbed entries. See text for further details.
• Figure 4: Overview of the identification possibilities in each setting. In the infinite horizon, we provide exact and numerical estimation procedures for one parameter given the others; with $B=R=\mathop{\mathrm{I}}\nolimits$, we present $A$ given $Q$ and $Q$ given $A$. In the finite horizon, we can leverage terminal conditions to estimate both $A$ and $Q$ (if $Q_T=Q$) simultaneously. In the infinite horizon with both $x_t$ and $u_t$ partially observed, and given $B=R=\mathop{\mathrm{I}}\nolimits$, we can estimate $A$ and then estimate $Q$.

Theorems & Definitions (10)

• Definition 1: Identifiability, LehmCase98
• Proposition 1
• proof
• Proposition 2
• proof
• Proposition 3
• Definition 2
• Proposition 4
• proof
• Remark : Closed-loop identification