Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Fully Data-Driven Value Iteration for Stochastic LQR: Convergence, Robustness and Stability

Leilei Cui, Zhong-Ping Jiang, Petter N. Kolm, Grégoire G. Macqueron

TL;DR

The paper addresses data-driven, model-free control for discrete-time stochastic LQR with fully unknown dynamics and costs. It develops a rigorous VI framework, proving global exponential stability for exact VI and small-disturbance ISS for inexact VI, and introduces a fully data-driven ADP algorithm, R-LSVI, that learns the Hamiltonian from input-state data off-policy without requiring a stabilizing initial policy. The authors establish convergence guarantees and demonstrate robustness to disturbances and non-quadratic costs, with empirical validation on data center cooling and dynamic portfolio allocation showing stable, convergent, and adaptable performance. This work enables practical, robust data-driven control in noisy environments, eliminating the need for system identification while providing theoretical guarantees and real-world applicability in engineering and finance.

Abstract

Unlike traditional model-based reinforcement learning approaches that estimate system parameters from data, non-model-based data-driven control learns the optimal policy directly from input-state data without any intermediate model identification. Although this direct reinforcement learning approach offers increased adaptability and resilience to model misspecification, its reliance on raw data leaves it vulnerable to system noise and disturbances that may undermine convergence, robustness, and stability. In this article, we establish the convergence, robustness, and stability of value iteration (VI) for data-driven control of stochastic linear quadratic (LQ) systems in discrete-time with entirely unknown dynamics and cost. Our contributions are three-fold. First, we prove that VI is globally exponentially stable for any positive semidefinite initial value matrix in noise-free settings, thereby significantly relaxing restrictive assumptions on initial value functions in existing literature. Second, we extend our analysis to settings with external disturbances, proving that VI maintains small-disturbance input-to-state stability (ISS) and converges within a small neighborhood of the optimal solution when disturbances are sufficiently small. Third, we propose a new non-model-based robust adaptive dynamic programming (ADP) algorithm for adaptive optimal controller design, which, unlike existing procedures, requires no prior knowledge of an initial admissible control policy. Numerical experiments on a ``data center cooling'' problem demonstrate the convergence and stability of the algorithm compared to established methods, highlighting its robustness and adaptability for data-driven control in noisy environments. Finally, we apply the method to dynamic portfolio allocation, demonstrating its practical relevance outside traditional control tasks.

A Fully Data-Driven Value Iteration for Stochastic LQR: Convergence, Robustness and Stability

TL;DR

The paper addresses data-driven, model-free control for discrete-time stochastic LQR with fully unknown dynamics and costs. It develops a rigorous VI framework, proving global exponential stability for exact VI and small-disturbance ISS for inexact VI, and introduces a fully data-driven ADP algorithm, R-LSVI, that learns the Hamiltonian from input-state data off-policy without requiring a stabilizing initial policy. The authors establish convergence guarantees and demonstrate robustness to disturbances and non-quadratic costs, with empirical validation on data center cooling and dynamic portfolio allocation showing stable, convergent, and adaptable performance. This work enables practical, robust data-driven control in noisy environments, eliminating the need for system identification while providing theoretical guarantees and real-world applicability in engineering and finance.

Abstract

Unlike traditional model-based reinforcement learning approaches that estimate system parameters from data, non-model-based data-driven control learns the optimal policy directly from input-state data without any intermediate model identification. Although this direct reinforcement learning approach offers increased adaptability and resilience to model misspecification, its reliance on raw data leaves it vulnerable to system noise and disturbances that may undermine convergence, robustness, and stability. In this article, we establish the convergence, robustness, and stability of value iteration (VI) for data-driven control of stochastic linear quadratic (LQ) systems in discrete-time with entirely unknown dynamics and cost. Our contributions are three-fold. First, we prove that VI is globally exponentially stable for any positive semidefinite initial value matrix in noise-free settings, thereby significantly relaxing restrictive assumptions on initial value functions in existing literature. Second, we extend our analysis to settings with external disturbances, proving that VI maintains small-disturbance input-to-state stability (ISS) and converges within a small neighborhood of the optimal solution when disturbances are sufficiently small. Third, we propose a new non-model-based robust adaptive dynamic programming (ADP) algorithm for adaptive optimal controller design, which, unlike existing procedures, requires no prior knowledge of an initial admissible control policy. Numerical experiments on a ``data center cooling'' problem demonstrate the convergence and stability of the algorithm compared to established methods, highlighting its robustness and adaptability for data-driven control in noisy environments. Finally, we apply the method to dynamic portfolio allocation, demonstrating its practical relevance outside traditional control tasks.
Paper Structure (23 sections, 21 theorems, 112 equations, 4 figures, 1 algorithm)

This paper contains 23 sections, 21 theorems, 112 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries and Problem Formulation
  3. Exponential and Input-to-State Stability
  4. The Stochastic Linear Quadratic Regulator Problem in Discrete-Time
  5. Problem Formulation
  6. Main Contributions
  7. Robust Least-Squares VI
  8. Convergence Analysis
  9. Convergence of the Exact VI
  10. Convergence of the Inexact VI
  11. Convergence of R-LSVI
  12. Empirical Results
  13. Convergence Analysis
  14. Adaptivity under Non-Quadratic Costs
  15. An Application to Dynamic Portfolio Allocation
  16. ...and 8 more sections

Key Result

Theorem 1

Under Assumption assum:hyp_control, for Procedure prd:exact_VI it holds

Figures (4)

  • Figure 2: Convergence and stability of the nominal control (PI and VI), O-LSPI, LSPI, R-LSVI (with and without discount) and policy gradient algorithms on the data center cooling benchmark problem as the sample size increases. (Left) Cost relative error. Dashed lines represent the median relative error, with the shaded region covering the $25^{th}$ to $75^{th}$ percentiles, estimated from $100$ trajectories. (Right) Frequency of stabilizing controllers found by the algorithms. Only costs corresponding to stable control gains are plotted in the left panel.
  • Figure 4: Convergence and stability of the nominal control (VI and PI), LSPI, O-LSPI, R-LSVI and policy gradient algorithms on the data center cooling benchmark problem with non-quadratic cost. (Left) Costs as the exponent $\alpha$ varies. Dashed lines represent the median relative error, with the shaded region covering the $25^{th}$ to $75^{th}$ percentiles, estimated from $100$ trajectories. (Right) Frequency of stabilizing controllers found by the algorithms. Only costs corresponding to stable control gains are plotted in the left panel.
  • Figure 6: Convergence and stability of the nominal VI, O-LSPI and R-LSVI algorithms on the dynamic portfolio allocation problem as the sample size increases. (Left) Cost relative error. Dashed lines represent the median relative error, with the shaded region covering the $25^{th}$ to $75^{th}$ percentiles, estimated from $100$ trajectories. (Right) Frequency of stabilizing controllers found by the algorithms. Only costs corresponding to stable control gains are plotted in the left panel.
  • Figure 8: Convergence and stability of the nominal VI, O-LSPI and R-LSVI algorithms on the dynamic portfolio allocation problem with non-quadratic cost. (Left) Costs as the exponent $\alpha$ varies. Dashed lines represent the median relative error, with the shaded region covering the $25^{th}$ to $75^{th}$ percentiles, estimated from $100$ trajectories. (Right) Frequency of stabilizing controllers found by the algorithms. Only costs corresponding to stable control gains are plotted in the left panel.

Theorems & Definitions (48)

  • Definition 1
  • Definition 2
  • Definition 3
  • Theorem 1: Convergence of Exact VI
  • Remark 1
  • Proposition 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Remark 5
  • ...and 38 more