Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Confusing Instance Principle for Online Linear Quadratic Control

Waris Radji, Odalric-Ambrym Maillard

TL;DR

This paper introduces the Confusing Instance (CI) principle as a new lens for exploration in online Linear Quadratic Regulation with unknown dynamics. It extends the Minimum Empirical Divergence (MED) framework from discrete settings to continuous LQR via MED-LQ, combining rank-one perturbations, stability constraints, and an empirical-divergence objective to drive exploration toward informative model perturbations. MED-LQ avoids confidence-bound based methods, leverages explicit LQR structure and Lyapunov stability, and demonstrates competitive performance on classical control benchmarks and industrial-like tasks, including auto-stabilization scenarios. The work lays a theoretical and algorithmic foundation for CI-based exploration in large-scale MDPs and suggests future directions for formal regret analysis and deep RL extensions with CI-guided exploration.

Abstract

We revisit the problem of controlling linear systems with quadratic cost under unknown dynamics with model-based reinforcement learning. Traditional methods like Optimism in the Face of Uncertainty and Thompson Sampling, rooted in multi-armed bandits (MABs), face practical limitations. In contrast, we propose an alternative based on the Confusing Instance (CI) principle, which underpins regret lower bounds in MABs and discrete Markov Decision Processes (MDPs) and is central to the Minimum Empirical Divergence (MED) family of algorithms, known for their asymptotic optimality in various settings. By leveraging the structure of LQR policies along with sensitivity and stability analysis, we develop MED-LQ. This novel control strategy extends the principles of CI and MED beyond small-scale settings. Our benchmarks on a comprehensive control suite demonstrate that MED-LQ achieves competitive performance in various scenarios while highlighting its potential for broader applications in large-scale MDPs.

The Confusing Instance Principle for Online Linear Quadratic Control

TL;DR

This paper introduces the Confusing Instance (CI) principle as a new lens for exploration in online Linear Quadratic Regulation with unknown dynamics. It extends the Minimum Empirical Divergence (MED) framework from discrete settings to continuous LQR via MED-LQ, combining rank-one perturbations, stability constraints, and an empirical-divergence objective to drive exploration toward informative model perturbations. MED-LQ avoids confidence-bound based methods, leverages explicit LQR structure and Lyapunov stability, and demonstrates competitive performance on classical control benchmarks and industrial-like tasks, including auto-stabilization scenarios. The work lays a theoretical and algorithmic foundation for CI-based exploration in large-scale MDPs and suggests future directions for formal regret analysis and deep RL extensions with CI-guided exploration.

Abstract

We revisit the problem of controlling linear systems with quadratic cost under unknown dynamics with model-based reinforcement learning. Traditional methods like Optimism in the Face of Uncertainty and Thompson Sampling, rooted in multi-armed bandits (MABs), face practical limitations. In contrast, we propose an alternative based on the Confusing Instance (CI) principle, which underpins regret lower bounds in MABs and discrete Markov Decision Processes (MDPs) and is central to the Minimum Empirical Divergence (MED) family of algorithms, known for their asymptotic optimality in various settings. By leveraging the structure of LQR policies along with sensitivity and stability analysis, we develop MED-LQ. This novel control strategy extends the principles of CI and MED beyond small-scale settings. Our benchmarks on a comprehensive control suite demonstrate that MED-LQ achieves competitive performance in various scenarios while highlighting its potential for broader applications in large-scale MDPs.
Paper Structure (30 sections, 3 theorems, 21 equations, 3 figures, 1 algorithm)

This paper contains 30 sections, 3 theorems, 21 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Research gap.
  3. From MABs to large MDPs.
  4. Outline and contributions.
  5. Setup and Background material
  6. The optimal control problem.
  7. The learning problem.
  8. Policy evaluation.
  9. Optimal MABs strategies.
  10. Efficient Confusing Instance Search for LQR
  11. Intuition.
  12. The Challenge of Approaching the Most Confusing Instance
  13. Fast Approximate Solution for Small Perturbed Systems
  14. Towards Minimum Empirical Divergence Strategies for Online LQR
  15. The MED-LQ Algorithm
  16. ...and 15 more sections

Key Result

Proposition 1

Given a gain $K$ that is stabilizing for the two systems $\Theta$ and $\widetilde{\Theta}$, and assuming both systems share the same covariance matrix $\Omega$, the asymptotic per-step expected log-likelihood under the two systems is where $\tau_T$ denotes a trajectory of length $T$ from $\pi_K$ and the stationary distribution $\Sigma_K(\Theta)$ induced by $K$ satisfies a discrete-time Lyapunov e

Figures (3)

  • Figure 1: Optimization landscape of the objective $\mathcal{L}(\alpha)$ on two Inverted Pendulum system barto1983neuronlike parametrized by the mass and the length of the pendulum. $\Theta$ is parametrized by (.1, .4) and $\Theta'$ by (.3, 1.). Red arrows indicate the Newton steps taken during the optimization process.
  • Figure 2: Performance comparison of MED-LQ under two distinct initialization scenarios.
  • Figure 3: Study of the sample size on the Inverted Pendulum environment.

Theorems & Definitions (7)

  • Proposition 1: Asymptotic per-step expected log-likelihood ratio for LQR
  • Proposition 2: Sub-optimality cost refinement
  • proof
  • Proposition 3: Sub-optimality cost refinement under small perturbations
  • Definition 4: Minimum Empirical Divergence coefficients for LQR
  • proof : Proof of Proposition \ref{['prop:elr']}.
  • proof : Proof of Proposition \ref{['prop:sub']}