Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Foundations of Safe Online Reinforcement Learning in the Linear Quadratic Regulator: $\sqrt{T}$-Regret

Benjamin Schiffer, Lucas Janson

TL;DR

The paper tackles safe online reinforcement learning for a one-dimensional LQR with unknown dynamics, introducing a safe algorithm that achieves $\tilde{O}_T(\sqrt{T})$ regret relative to a baseline of truncated linear controllers. The approach combines a warm-up exploration, regularized estimation of the unknown dynamics, and regime-dependent control: near-unconstrained control when noise is small and truncated-certainty-equivalence control when noise is large, with safety enforced via dynamically adjusted bounds. Key theoretical contributions include proving high-probability safety and square-root regret, plus Lipschitz-continuity properties of the truncated baseline that enable the analysis. This work strengthens regret guarantees in safety-constrained RL and provides a concrete, analyzable baseline (truncated linear controllers) that is well-suited to safety constraints in linear systems, with implications for practical safe learning in constrained control tasks.

Abstract

Understanding how to efficiently learn while adhering to safety constraints is essential for using online reinforcement learning in practical applications. However, proving rigorous regret bounds for safety-constrained reinforcement learning is difficult due to the complex interaction between safety, exploration, and exploitation. In this work, we seek to establish foundations for safety-constrained reinforcement learning by studying the canonical problem of controlling a one-dimensional linear dynamical system with unknown dynamics. We study the safety-constrained version of this problem, where the state must with high probability stay within a safe region, and we provide the first safe algorithm that achieves regret of $\tilde{O}_T(\sqrt{T})$. Furthermore, the regret is with respect to the baseline of truncated linear controllers, a natural baseline of non-linear controllers that are well-suited for safety-constrained linear systems. In addition to introducing this new baseline, we also prove several desirable continuity properties of the optimal controller in this baseline. In showing our main result, we prove that whenever the constraints impact the optimal controller, the non-linearity of our controller class leads to a faster rate of learning than in the unconstrained setting.

Foundations of Safe Online Reinforcement Learning in the Linear Quadratic Regulator: $\sqrt{T}$-Regret

TL;DR

The paper tackles safe online reinforcement learning for a one-dimensional LQR with unknown dynamics, introducing a safe algorithm that achieves regret relative to a baseline of truncated linear controllers. The approach combines a warm-up exploration, regularized estimation of the unknown dynamics, and regime-dependent control: near-unconstrained control when noise is small and truncated-certainty-equivalence control when noise is large, with safety enforced via dynamically adjusted bounds. Key theoretical contributions include proving high-probability safety and square-root regret, plus Lipschitz-continuity properties of the truncated baseline that enable the analysis. This work strengthens regret guarantees in safety-constrained RL and provides a concrete, analyzable baseline (truncated linear controllers) that is well-suited to safety constraints in linear systems, with implications for practical safe learning in constrained control tasks.

Abstract

Understanding how to efficiently learn while adhering to safety constraints is essential for using online reinforcement learning in practical applications. However, proving rigorous regret bounds for safety-constrained reinforcement learning is difficult due to the complex interaction between safety, exploration, and exploitation. In this work, we seek to establish foundations for safety-constrained reinforcement learning by studying the canonical problem of controlling a one-dimensional linear dynamical system with unknown dynamics. We study the safety-constrained version of this problem, where the state must with high probability stay within a safe region, and we provide the first safe algorithm that achieves regret of . Furthermore, the regret is with respect to the baseline of truncated linear controllers, a natural baseline of non-linear controllers that are well-suited for safety-constrained linear systems. In addition to introducing this new baseline, we also prove several desirable continuity properties of the optimal controller in this baseline. In showing our main result, we prove that whenever the constraints impact the optimal controller, the non-linearity of our controller class leads to a faster rate of learning than in the unconstrained setting.

Paper Structure

This paper contains 67 sections, 56 theorems, 337 equations, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Setting and Motivation
  4. Our Contribution
  5. Related Work
  6. Preliminaries
  7. Dynamics and Cost
  8. Safety Constraints
  9. Baseline Class
  10. Initial Uncertainty
  11. Problem Statement
  12. Theoretical Results
  13. Proof Sketch of Theorem \ref{['arxiv2:trunc_thm']}
  14. Algorithm \ref{['arxiv2:alg:cap3']} Intuition
  15. Sufficiently small noise case
  16. ...and 52 more sections

Key Result

Theorem 1

For safety-constrained one-dimensional LQR with unknown dynamics and any sub-gaussian noise distribution, there exists an algorithm that with high probability is safe and has regret of $\tilde{O}_T(\sqrt{T})$ compared to the best truncated linear controller with known dynamics.

Theorems & Definitions (95)

  • Theorem 1: Informal
  • Definition 1
  • Theorem 2
  • Lemma 1
  • Lemma 2
  • proof
  • Lemma 3
  • Lemma 4
  • Definition 2
  • Lemma 5
  • ...and 85 more