Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Stabilizing Policies via an Unstable Subspace Representation

Leonardo F. Toso, Lintao Ye, James Anderson

TL;DR

This work addresses learning to stabilize unknown high-dimensional LTI systems by a two-phase, model-free approach that first learns the left unstable subspace and then performs discounted LQR on the learned low-dimensional subspace. By focusing on the unstable modes, the method reduces the effective control dimension and sample complexity, achieving a non-asymptotic guarantee that scales as $\tilde{O}(\ell^2 d_U)$ (plus an $\mathcal{O}(d_X)$ term) rather than $\tilde{O}(d_X^2 d_U)$ for the full-space problem. Key components include finite-sample guarantees for estimating the left unstable representation from adjoint data, a zeroth-order gradient estimator for the low-dimensional problem, and an explicit discount-annealing scheme that leads to a stabilizing controller $K=\theta\widehat{\Phi}^T$ with high probability. The proposed approach accommodates non-diagonalizable dynamics via Jordan form and yields substantial practical gains in instability-driven dimensionality reduction, with demonstrated improvements in sample efficiency in numerical experiments.

Abstract

We study the problem of learning to stabilize (LTS) a linear time-invariant (LTI) system. Policy gradient (PG) methods for control assume access to an initial stabilizing policy. However, designing such a policy for an unknown system is one of the most fundamental problems in control, and it may be as hard as learning the optimal policy itself. Existing work on the LTS problem requires large data as it scales quadratically with the ambient dimension. We propose a two-phase approach that first learns the left unstable subspace of the system and then solves a series of discounted linear quadratic regulator (LQR) problems on the learned unstable subspace, targeting to stabilize only the system's unstable dynamics and reduce the effective dimension of the control space. We provide non-asymptotic guarantees for both phases and demonstrate that operating on the unstable subspace reduces sample complexity. In particular, when the number of unstable modes is much smaller than the state dimension, our analysis reveals that LTS on the unstable subspace substantially speeds up the stabilization process. Numerical experiments are provided to support this sample complexity reduction achieved by our approach.

Learning Stabilizing Policies via an Unstable Subspace Representation

TL;DR

This work addresses learning to stabilize unknown high-dimensional LTI systems by a two-phase, model-free approach that first learns the left unstable subspace and then performs discounted LQR on the learned low-dimensional subspace. By focusing on the unstable modes, the method reduces the effective control dimension and sample complexity, achieving a non-asymptotic guarantee that scales as (plus an term) rather than for the full-space problem. Key components include finite-sample guarantees for estimating the left unstable representation from adjoint data, a zeroth-order gradient estimator for the low-dimensional problem, and an explicit discount-annealing scheme that leads to a stabilizing controller with high probability. The proposed approach accommodates non-diagonalizable dynamics via Jordan form and yields substantial practical gains in instability-driven dimensionality reduction, with demonstrated improvements in sample efficiency in numerical experiments.

Abstract

We study the problem of learning to stabilize (LTS) a linear time-invariant (LTI) system. Policy gradient (PG) methods for control assume access to an initial stabilizing policy. However, designing such a policy for an unknown system is one of the most fundamental problems in control, and it may be as hard as learning the optimal policy itself. Existing work on the LTS problem requires large data as it scales quadratically with the ambient dimension. We propose a two-phase approach that first learns the left unstable subspace of the system and then solves a series of discounted linear quadratic regulator (LQR) problems on the learned unstable subspace, targeting to stabilize only the system's unstable dynamics and reduce the effective dimension of the control space. We provide non-asymptotic guarantees for both phases and demonstrate that operating on the unstable subspace reduces sample complexity. In particular, when the number of unstable modes is much smaller than the state dimension, our analysis reveals that LTS on the unstable subspace substantially speeds up the stabilization process. Numerical experiments are provided to support this sample complexity reduction achieved by our approach.
Paper Structure (30 sections, 24 theorems, 112 equations, 6 figures, 1 algorithm)

This paper contains 30 sections, 24 theorems, 112 equations, 6 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Notation
  5. Problem Formulation
  6. Discounted Linear Quadratic Regulator Problem
  7. Stabilizing Only the Unstable Modes
  8. Low-Dimensional Discounted LQR Problem
  9. Learning the Left Unstable Representation
  10. Learning to Stabilize on the Unstable Subspace
  11. Gradient and Cost Estimation
  12. Discounted LQR on the Unstable Subspace
  13. Sample Complexity Analysis
  14. Numerical Validation
  15. Conclusions and Future Work
  16. ...and 15 more sections

Key Result

Lemma 2.1

Given high-dimensional and low-dimensional stabilizing controllers $K, K^\prime \in \mathcal{S}^\gamma_K$ and $\theta, \theta^\prime \in \mathcal{S}^\gamma_\theta$, respectively. It holds that $\|\nabla J^\gamma(K)\| \leq \phi, \|\theta\| \leq \nu_\theta$, and

Figures (6)

  • Figure 1: Workflow for learning to stabilize (LTS) a high-dimensional (HD) discrete-time LTI system on its low-dimensional (LD) unstable subspace.
  • Figure 2: Subspace distance $d(\widehat{\Phi},\Phi)$ with respect to data $(T)$.
  • Figure 3: Closed-loop spectral radius w.r.t. iterations.
  • Figure 4: Discount factor w.r.t. iterations.
  • Figure 5: Closed-loop spectral radius (left) and discount factor (right) w.r.t the iteration count.
  • ...and 1 more figures

Theorems & Definitions (35)

  • Remark 2.1
  • Definition 2.1
  • Lemma 2.1
  • Lemma 2.2
  • Remark 2.2
  • Definition 2.2
  • Lemma 3.1
  • Lemma 3.2
  • Theorem 3.1
  • Remark 3.1
  • ...and 25 more