The Role of Identification in Data-driven Policy Iteration: A System Theoretic Study

TL;DR

The paper analyzes indirect and direct data-driven policy iteration for unknown discrete-time LQR problems by treating the learning and control loops as a coupled dynamical system. Indirect PI (IPI) combines online recursive identification (RLS) with classical model-based PI and is shown to converge under local persistency of excitation, with favorable sample efficiency and robustness properties. Direct PI (DPI) operates without explicit model identification, using data-driven policy evaluation and improvement; it requires stronger persistency and larger data budgets but offers a direct data-centric alternative. A detailed comparison reveals that IPI generally achieves faster convergence with fewer samples, while DPI can match performance under sufficient excitation but with higher data demands; simulations corroborate the theoretical findings. The study provides guidance on episode design, excitation strategies, and stability considerations, and outlines future work for noisy settings and adaptive excitation.

Abstract

The goal of this article is to study fundamental mechanisms behind so-called indirect and direct data-driven control for unknown systems. Specifically, we consider policy iteration applied to the linear quadratic regulator problem. Two iterative procedures, where data collected from the system are repeatedly used to compute new estimates of the desired optimal controller, are considered. In indirect policy iteration, data are used to obtain an updated model estimate through a recursive identification scheme, which is used in a certainty-equivalent fashion to perform the classic policy iteration update. By casting the concurrent model identification and control design as a feedback interconnection between two algorithmic systems, we provide a closed-loop analysis that shows convergence and robustness properties for arbitrary levels of excitation in the data. In direct policy iteration, data are used to approximate the value function and design the associated controller without requiring the intermediate identification step. After proposing an extension to a recently proposed scheme that overcomes potential identifiability issues, we establish under which conditions this procedure is guaranteed to deliver the optimal controller. Based on these analyses we are able to compare the strengths and limitations of the two approaches, highlighting aspects such as the required samples, convergence properties, and excitation requirement. Simulations are also provided to illustrate the results.

Figures (6)

• Figure 1: Illustration of episode indices $i$ and time indices $t$.
• Figure 2: Concurrent identification and policy iteration scheme.
• Figure 3: Graphical representation of sequence properties.
• Figure 4: Convergence analysis of coupled dynamical systems.
• Figure 5: Comparison of DPI and IPI with different episode lengths $\tau_\mathrm{IPI}$ and minimum length $\tau_\mathrm{DPI}$ (the symbol $\tau$ in the $x$-axis label denotes the length of the episode in the respective approach).

Theorems & Definitions (27)

• Theorem 1
• Remark 1
• Remark 2
• Definition 1
• Definition 2
• Theorem 2
• Remark 3
• Remark 4
• Definition 3
• Definition 4