Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Output Feedback Stabilization of Linear Systems via Policy Gradient Methods

Ankang Zhang, Ming Chi, Xiaoling Wang, Lintao Ye

TL;DR

This work tackles stabilizing unknown, partially observable discrete-time LTI systems using model-free policy gradient methods applied to a static output feedback form $u_t=-K y_t$. By introducing a discount-based framework and leveraging zeroth-order gradient estimates from system rollouts, the authors show convergence to $\\epsilon$-stationary points in the SOF policy space and provide explicit sample-complexity bounds for stabilization. The approach yields a practical, data-efficient procedure that gradually increases the discount factor toward 1, enabling stabilization of the original system without full model knowledge. Theoretical guarantees are complemented by numerical experiments on synthetic unstable dynamics and a cart-pole example, highlighting the method's applicability to partially observable, unknown systems and its potential for real-world control tasks.

Abstract

Stabilizing a dynamical system is a fundamental problem that serves as a cornerstone for many complex tasks in the field of control systems. The problem becomes challenging when the system model is unknown. Among the Reinforcement Learning (RL) algorithms that have been successfully applied to solve problems pertaining to unknown linear dynamical systems, the policy gradient (PG) method stands out due to its ease of implementation and can solve the problem in a model-free manner. However, most of the existing works on PG methods for unknown linear dynamical systems assume full-state feedback. In this paper, we take a step towards model-free learning for partially observable linear dynamical systems with output feedback and focus on the fundamental stabilization problem of the system. We propose an algorithmic framework that stretches the boundary of PG methods to the problem without global convergence guarantees. We show that by leveraging zeroth-order PG update based on system trajectories and its convergence to stationary points, the proposed algorithms return a stabilizing output feedback policy for discrete-time linear dynamical systems. We also explicitly characterize the sample complexity of our algorithm and verify the effectiveness of the algorithm using numerical examples.

Output Feedback Stabilization of Linear Systems via Policy Gradient Methods

TL;DR

This work tackles stabilizing unknown, partially observable discrete-time LTI systems using model-free policy gradient methods applied to a static output feedback form . By introducing a discount-based framework and leveraging zeroth-order gradient estimates from system rollouts, the authors show convergence to -stationary points in the SOF policy space and provide explicit sample-complexity bounds for stabilization. The approach yields a practical, data-efficient procedure that gradually increases the discount factor toward 1, enabling stabilization of the original system without full model knowledge. Theoretical guarantees are complemented by numerical experiments on synthetic unstable dynamics and a cart-pole example, highlighting the method's applicability to partially observable, unknown systems and its potential for real-world control tasks.

Abstract

Stabilizing a dynamical system is a fundamental problem that serves as a cornerstone for many complex tasks in the field of control systems. The problem becomes challenging when the system model is unknown. Among the Reinforcement Learning (RL) algorithms that have been successfully applied to solve problems pertaining to unknown linear dynamical systems, the policy gradient (PG) method stands out due to its ease of implementation and can solve the problem in a model-free manner. However, most of the existing works on PG methods for unknown linear dynamical systems assume full-state feedback. In this paper, we take a step towards model-free learning for partially observable linear dynamical systems with output feedback and focus on the fundamental stabilization problem of the system. We propose an algorithmic framework that stretches the boundary of PG methods to the problem without global convergence guarantees. We show that by leveraging zeroth-order PG update based on system trajectories and its convergence to stationary points, the proposed algorithms return a stabilizing output feedback policy for discrete-time linear dynamical systems. We also explicitly characterize the sample complexity of our algorithm and verify the effectiveness of the algorithm using numerical examples.
Paper Structure (27 sections, 16 theorems, 99 equations, 2 figures)

This paper contains 27 sections, 16 theorems, 99 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Problem Formulation and Preliminaries
  3. Problem Formulation
  4. Discount Method for Output Feedback Control
  5. Algorithm Design
  6. Gradient Estimation
  7. Convergence Properties of PG Method
  8. Learning to Stabilize Systems by Discount Methods
  9. Algorithm Analysis
  10. Convergence Analysis
  11. Sample Complexity
  12. Simulation Results
  13. Numerical Example
  14. Application to Cart-Pole System
  15. Conclusion
  16. ...and 12 more sections

Key Result

Lemma 1

cohen2019learning Suppose that an SF policy $F$ satisfies $F\in\{F~|~ J(F)\leq \nu\}$, then $F$ is $(\kappa,\varrho)$-strongly stable with $\kappa=\sqrt{\nu/\ell_0}$ and $\varrho=1/2\kappa^2$.

Figures (2)

  • Figure 1: Simulation results of Algorithm \ref{['alg:learning_stabilize']} applied to the system \ref{['eq-simulation']} versus iterations.
  • Figure 2: Simulation results of Algorithm \ref{['alg:learning_stabilize']} applied to the cart-pole system \ref{['eq:discrete cartpole']} versus iterations.

Theorems & Definitions (31)

  • Definition 1
  • Definition 2
  • Definition 3
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Theorem 1
  • proof
  • ...and 21 more