Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the stability of Lipschitz continuous control problems and its application to reinforcement learning

Namkyeong Cho, Yeoneung Kim

TL;DR

We address the stability of the Hamilton–Jacobi–Bellman equation in model-free reinforcement learning for Lipschitz-continuous control problems and connect Lipschitz controls with viscosity-solution theory. The authors establish stability and convergence properties of the Lipschitz-parameterized value function $Q^L(x,a)$, derive rates under structural assumptions, and propose a generalized HJB-based Q-learning framework that includes a family of constrained controls $\mathcal{A}_p^L$ with corresponding $Q_p^L$. They prove that $Q^L$ converges locally uniformly to the classical $Q$ as $L\to\infty$ and provide rates when additional regularity holds, alongside a practical algorithm ($p$-HJDQN) and empirical results on standard continuous-control benchmarks. The results highlight the interplay between Lipschitz constraints, the choice of norm $p$, and convergence behavior, offering a principled route to stable, viscosity-physics–inspired Q-learning with tunable control regularity.

Abstract

We address the crucial yet underexplored stability properties of the Hamilton--Jacobi--Bellman (HJB) equation in model-free reinforcement learning contexts, specifically for Lipschitz continuous optimal control problems. We bridge the gap between Lipschitz continuous optimal control problems and classical optimal control problems in the viscosity solutions framework, offering new insights into the stability of the value function of Lipschitz continuous optimal control problems. By introducing structural assumptions on the dynamics and reward functions, we further study the rate of convergence of value functions. Moreover, we introduce a generalized framework for Lipschitz continuous control problems that incorporates the original problem and leverage it to propose a new HJB-based reinforcement learning algorithm. The stability properties and performance of the proposed method are tested with well-known benchmark examples in comparison with existing approaches.

On the stability of Lipschitz continuous control problems and its application to reinforcement learning

TL;DR

We address the stability of the Hamilton–Jacobi–Bellman equation in model-free reinforcement learning for Lipschitz-continuous control problems and connect Lipschitz controls with viscosity-solution theory. The authors establish stability and convergence properties of the Lipschitz-parameterized value function , derive rates under structural assumptions, and propose a generalized HJB-based Q-learning framework that includes a family of constrained controls with corresponding . They prove that converges locally uniformly to the classical as and provide rates when additional regularity holds, alongside a practical algorithm (-HJDQN) and empirical results on standard continuous-control benchmarks. The results highlight the interplay between Lipschitz constraints, the choice of norm , and convergence behavior, offering a principled route to stable, viscosity-physics–inspired Q-learning with tunable control regularity.

Abstract

We address the crucial yet underexplored stability properties of the Hamilton--Jacobi--Bellman (HJB) equation in model-free reinforcement learning contexts, specifically for Lipschitz continuous optimal control problems. We bridge the gap between Lipschitz continuous optimal control problems and classical optimal control problems in the viscosity solutions framework, offering new insights into the stability of the value function of Lipschitz continuous optimal control problems. By introducing structural assumptions on the dynamics and reward functions, we further study the rate of convergence of value functions. Moreover, we introduce a generalized framework for Lipschitz continuous control problems that incorporates the original problem and leverage it to propose a new HJB-based reinforcement learning algorithm. The stability properties and performance of the proposed method are tested with well-known benchmark examples in comparison with existing approaches.
Paper Structure (19 sections, 9 theorems, 132 equations, 7 figures, 1 table, 1 algorithm)

This paper contains 19 sections, 9 theorems, 132 equations, 7 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Setup and Preliminaries
  3. Some quantitative properties of $Q^L$
  4. Lipschitz Regularity
  5. Error Bounds of $Q^{L+\ell}-Q^{L}$ for $L,\ell \geq 0$.
  6. Convergence of $Q^L(x,a)$ to $Q(x)$
  7. General convergence result
  8. Rate of convergence for some special cases
  9. Generalized Hamilton--Jacobi based Q-learning
  10. Demonstration of optimal action
  11. Convergence result
  12. Numerical experiment
  13. Proposed algorithm
  14. Numerical analysis on convergence result
  15. Numerical analysis on different values of $1\leq p\leq \infty$
  16. ...and 4 more sections

Key Result

Proposition 2.2

Under Assumption ass:naive, the unique viscosity solution $Q$ to is Lipschitz continuous if $\gamma> \|f\|_{\text{Lip}(\mathbb{R}^n\times \mathbb{R}^m)}$.

Figures (7)

  • Figure 1: Plot for the Table \ref{['table:Q_diff']}
  • Figure 2: Learning curves for different tasks.
  • Figure 3: Action trajectories obtained by different values of $p$ in HJDQN and DDPG for HalfCheetah-v2.
  • Figure 4: Action trajectories obtained by different values of $p$ in HJDQN and DDPG for Walker2d-v2.
  • Figure 5: Action trajectories obtained by HJDQN with $p=2$ and DDPG for 20D LQR problem.
  • ...and 2 more figures

Theorems & Definitions (18)

  • Proposition 2.2
  • proof
  • Lemma 3.1
  • proof
  • Lemma 3.2
  • proof
  • Definition 4.1
  • Theorem 4.2
  • proof
  • Proposition 4.4
  • ...and 8 more