Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Challenges in Model Agnostic Controller Learning for Unstable Systems

Mario Sznaier, Mustafa Bozdag

TL;DR

This paper addresses the risk that model-agnostic direct policy optimization can destabilize even simple unstable systems due to pole/zero cancellations that preserve input-output performance but destroy internal stability. It analyzes a minimal first-order example using Youla parameterization to show how interpolation constraints are essential to maintain internal stability, and demonstrates that unconstrained or nonlinear approaches can yield unstable closed-loop behavior. The authors discuss remedies—such as adding training noise, prestabilization, and data-driven coprime-factorization with gap-metric guarantees—and highlight the limitations of each, emphasizing the need for stability-aware, time-domain learning frameworks. The work suggests a shift toward learning stabilizing controllers from finite data via stability-centric loss functions, with implications for model-free control design in practice.

Abstract

Model agnostic controller learning, for instance by direct policy optimization, has been the object of renewed attention lately, since it avoids a computationally expensive system identification step. Indeed, direct policy search has been empirically shown to lead to optimal controllers in a number of cases of practical importance. However, to date, these empirical results have not been backed up with a comprehensive theoretical analysis for general problems. In this paper we use a simple example to show that direct policy optimization is not directly generalizable to other seemingly simple problems. In such cases, direct optimization of a performance index can lead to unstable pole/zero cancellations, resulting in the loss of internal stability and unbounded outputs in response to arbitrarily small perturbations. We conclude the paper by analyzing several alternatives to avoid this phenomenon, suggesting some new directions in direct control policy optimization.

Challenges in Model Agnostic Controller Learning for Unstable Systems

TL;DR

This paper addresses the risk that model-agnostic direct policy optimization can destabilize even simple unstable systems due to pole/zero cancellations that preserve input-output performance but destroy internal stability. It analyzes a minimal first-order example using Youla parameterization to show how interpolation constraints are essential to maintain internal stability, and demonstrates that unconstrained or nonlinear approaches can yield unstable closed-loop behavior. The authors discuss remedies—such as adding training noise, prestabilization, and data-driven coprime-factorization with gap-metric guarantees—and highlight the limitations of each, emphasizing the need for stability-aware, time-domain learning frameworks. The work suggests a shift toward learning stabilizing controllers from finite data via stability-centric loss functions, with implications for model-free control design in practice.

Abstract

Model agnostic controller learning, for instance by direct policy optimization, has been the object of renewed attention lately, since it avoids a computationally expensive system identification step. Indeed, direct policy search has been empirically shown to lead to optimal controllers in a number of cases of practical importance. However, to date, these empirical results have not been backed up with a comprehensive theoretical analysis for general problems. In this paper we use a simple example to show that direct policy optimization is not directly generalizable to other seemingly simple problems. In such cases, direct optimization of a performance index can lead to unstable pole/zero cancellations, resulting in the loss of internal stability and unbounded outputs in response to arbitrarily small perturbations. We conclude the paper by analyzing several alternatives to avoid this phenomenon, suggesting some new directions in direct control policy optimization.
Paper Structure (16 sections, 1 theorem, 47 equations, 10 figures)

This paper contains 16 sections, 1 theorem, 47 equations, 10 figures.

Table of Contents

  1. Introduction
  2. Notation and Definitions
  3. A simple first order example
  4. Optimization over all internally stabilizing controllers
  5. Input/Output optimization over LTI controllers
  6. General Optimization based nonlinear controllers
  7. Possible solutions to avoid losing internal stability and their limitations.
  8. Conclusions
  9. Control Theory background material
  10. Coprime Factorizations
  11. Inner Transfer Functions
  12. Youla Parameterization
  13. Youla parameterization in the time domain
  14. Parameterization of all achievable closed-loop maps.
  15. Internal Stability and the Interpolation Constraints
  16. ...and 1 more sections

Key Result

Theorem 1

Consider a finite dimensional nonlinear controller of the form where $\boldsymbol{\theta}_k \doteq $, ${\bm{u}}_{k-1} \doteq ^T$, ${\bm{e}}_{k} \doteq $, $m$ is the memory of the controller, and $f_u$ is continuous. Let $\Phi_{cl}$ denote the corresponding closed loop mapping from the input $r$ to the output sequence $\left \{^T \right\}$. If the controller eq: then the closed loop mapping $^T \t

Figures (10)

  • Figure 1: The closed loop is internally stable if all four mappings $rw^T \rightarrow ev^T$ are stable.
  • Figure 2: Closed loop for the simple example.
  • Figure 3: Closed loop responses for the controllers \ref{['eq:optimal']} and \ref{['eq:controller']}: (a) Tracking error; (b) Response to a random perturbation $w$.
  • Figure 4: A controller that minimizes $\frac{\|e\|_2}{|r_o|}$ cannot render the mapping $\left[r \quad w\right]^T \to \left[u \quad e\right]^T$ finite $\ell^\infty$ gain stable.
  • Figure 5: Illustration of Theorem \ref{['teo:mainlinf']}. (a) Control action for the neural net controller, with and without noise added. (b) A small perturbation $w$ leads to an unbounded output.
  • ...and 5 more figures

Theorems & Definitions (7)

  • Definition 1
  • Definition 2
  • Remark 1
  • Theorem 1
  • proof
  • Definition 3
  • Definition 4