Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analyzing the Impact of Computation in Adaptive Dynamic Programming for Stochastic LQR Problem

Wenhan Cao, Alexandre Capone, Sandra Hirche, Wei Pan

TL;DR

The paper tackles model-free adaptive dynamic programming for stochastic LQR, where stochastic integrals must be approximated from discrete-time samples using the Euler-Maruyama method with sampling period $h$. It reframes policy iteration as Newton's method for the generalized Riccati equation and derives an $O(h)$ per-iteration error bound, culminating in an overall convergence bound $\lim_{k\to\infty}\|\hat{P}_k-P^*\|_F \le \frac{\|C_s\|}{1-L_P} h$. Theoretical results are corroborated by a sensorimotor task, showing that both the value matrix $\hat{P}_\infty$ and the gain $\hat{K}_\infty$ converge to their optima at rate $O(h)$. The findings highlight a fundamental trade-off between sampling efficiency and control performance in data-driven stochastic ADP, with practical implications for real-time policy learning.

Abstract

Adaptive dynamic programming (ADP) for stochastic linear quadratic regulation (LQR) demands the precise computation of stochastic integrals during policy iteration (PI). In a fully model-free problem setting, this computation can only be approximated by state samples collected at discrete time points using computational methods such as the canonical Euler-Maruyama method. Our research reveals a critical phenomenon: the sampling period can significantly impact control performance. This impact is due to the fact that computational errors introduced in each step of PI can significantly affect the algorithm's convergence behavior, which in turn influences the resulting control policy. We draw a parallel between PI and Newton's method applied to the Ricatti equation to elucidate how the computation impacts control. In this light, the computational error in each PI step manifests itself as an extra error term in each step of Newton's method, with its upper bound proportional to the computational error. Furthermore, we demonstrate that the convergence rate for ADP in stochastic LQR problems using the Euler-Maruyama method is O(h), with h being the sampling period. A sensorimotor control task finally validates these theoretical findings.

Analyzing the Impact of Computation in Adaptive Dynamic Programming for Stochastic LQR Problem

TL;DR

The paper tackles model-free adaptive dynamic programming for stochastic LQR, where stochastic integrals must be approximated from discrete-time samples using the Euler-Maruyama method with sampling period . It reframes policy iteration as Newton's method for the generalized Riccati equation and derives an per-iteration error bound, culminating in an overall convergence bound . Theoretical results are corroborated by a sensorimotor task, showing that both the value matrix and the gain converge to their optima at rate . The findings highlight a fundamental trade-off between sampling efficiency and control performance in data-driven stochastic ADP, with practical implications for real-time policy learning.

Abstract

Adaptive dynamic programming (ADP) for stochastic linear quadratic regulation (LQR) demands the precise computation of stochastic integrals during policy iteration (PI). In a fully model-free problem setting, this computation can only be approximated by state samples collected at discrete time points using computational methods such as the canonical Euler-Maruyama method. Our research reveals a critical phenomenon: the sampling period can significantly impact control performance. This impact is due to the fact that computational errors introduced in each step of PI can significantly affect the algorithm's convergence behavior, which in turn influences the resulting control policy. We draw a parallel between PI and Newton's method applied to the Ricatti equation to elucidate how the computation impacts control. In this light, the computational error in each PI step manifests itself as an extra error term in each step of Newton's method, with its upper bound proportional to the computational error. Furthermore, we demonstrate that the convergence rate for ADP in stochastic LQR problems using the Euler-Maruyama method is O(h), with h being the sampling period. A sensorimotor control task finally validates these theoretical findings.
Paper Structure (6 sections, 5 theorems, 30 equations, 2 figures)

This paper contains 6 sections, 5 theorems, 30 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Computational Error Analysis
  4. Convergence Analysis
  5. Numerical Results
  6. Conclusion and Discussion

Key Result

lemma 1

Suppose $X^{h}(t)$ is the discrete-time approximation of $X(t)$ through the Euler-Maruyama method with the time step $h$. If $\mu$ and $\sigma$ are bounded and Lipschitz continuous, then the computational error $|\mathbb{E}X(t) - \mathbb{E}X^{h}(t)|$ satisfies with $C>0$ being a constant number depending on the formulation of the integral.

Figures (2)

  • Figure 1: Expected cost $J_E$ for controllers, computed by the Euler-Maruyama method with varying sample period $h$ for a sensorimotor control task.
  • Figure 2: Convergence of PI solutions with varying sampling periods $h$. Left: $\|\hat{P}_{\infty} - P^*\|_F$ demonstrates a convergence rate of $O(h)$. Right: $\|\hat{K}_{\infty} - K^*\|_F$ demonstrates a convergence rate of $O(h)$.

Theorems & Definitions (13)

  • definition 1: Admissible matrix
  • remark 1
  • lemma 1: Computational error of Euler-Maruyama method bally1995eulerbally1996law1bally1996law2
  • remark 2
  • theorem 1
  • proof
  • lemma 2: PI as Newton's method
  • proof
  • lemma 3: Error Bound of Fixed Point Iteration urabe1956convergence
  • theorem 2: Convergence of PI with Computational Error
  • ...and 3 more