Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy Optimization in the Linear Quadratic Gaussian Problem: A Frequency Domain Perspective

Haoran Li, Xun Li, Yuan-Hua Ni, Xuebo Zhang

TL;DR

This work tackles global optimality certification for LQG policy optimization under direct controller parameterization. It introduces a frequency-domain, LFT-based optimality condition that yields a verifiable certificate and reveals why direct parameterizations can trap gradient methods in suboptimal stationary points. Building on this, the authors develop a gradient-based algorithm that operates in the infinite-dimensional $\mathcal{RH}_\infty$ space via Youla parameterization, with a proven global convergence guarantee and a data-driven extension for model-free scenarios. Numerical experiments illustrate the method's ability to escape suboptimal points and validate the data-driven estimation procedures for key operators. Collectively, the results provide both theoretical insight into the LQG optimization landscape and a practical pathway toward robust, data-driven controller synthesis.

Abstract

The Linear Quadratic Gaussian (LQG) problem is a classic and widely studied model in optimal control, providing a fundamental framework for designing controllers for linear systems subject to process and observation noises. In recent years, researchers have increasingly focused on directly parameterizing dynamic controllers and optimizing the LQG cost over the resulting parameterized set. However, this parameterization typically gives rise to a highly non-convex optimization landscape for the resulting parameterized LQG problem. To our knowledge, there is currently no general method for certifying the global optimality of candidate controller parameters in this setting. In this work, we address these gaps with the following contributions. First, we derive a necessary and sufficient condition for the global optimality of stationary points in a parameterized LQG problems. This condition reduces the verification of optimality to a test of the controllability and observability for a novel, specially constructed transfer function, yielding a precise and computationally tractable certificate. Furthermore, our condition provides a rigorous explanation for why traditional parameterizations can lead to suboptimal stationary points. Second, we elevate the controller parameter space from conventional finite-dimensional settings to the infinite-dimensional $\mathcal{RH}_\infty$ space and develop a gradient-based algorithm in this setting, for which we provide a theoretical analysis establishing global convergence. Finally, representative numerical experiments validate the theoretical findings and demonstrate the practical viability of the proposed approach. Additionally, the appendix section explores a data-driven extension to the model-free setting, where we outline a parameter estimation scheme and demonstrate its practical viability through numerical simulation.

Policy Optimization in the Linear Quadratic Gaussian Problem: A Frequency Domain Perspective

TL;DR

This work tackles global optimality certification for LQG policy optimization under direct controller parameterization. It introduces a frequency-domain, LFT-based optimality condition that yields a verifiable certificate and reveals why direct parameterizations can trap gradient methods in suboptimal stationary points. Building on this, the authors develop a gradient-based algorithm that operates in the infinite-dimensional space via Youla parameterization, with a proven global convergence guarantee and a data-driven extension for model-free scenarios. Numerical experiments illustrate the method's ability to escape suboptimal points and validate the data-driven estimation procedures for key operators. Collectively, the results provide both theoretical insight into the LQG optimization landscape and a practical pathway toward robust, data-driven controller synthesis.

Abstract

The Linear Quadratic Gaussian (LQG) problem is a classic and widely studied model in optimal control, providing a fundamental framework for designing controllers for linear systems subject to process and observation noises. In recent years, researchers have increasingly focused on directly parameterizing dynamic controllers and optimizing the LQG cost over the resulting parameterized set. However, this parameterization typically gives rise to a highly non-convex optimization landscape for the resulting parameterized LQG problem. To our knowledge, there is currently no general method for certifying the global optimality of candidate controller parameters in this setting. In this work, we address these gaps with the following contributions. First, we derive a necessary and sufficient condition for the global optimality of stationary points in a parameterized LQG problems. This condition reduces the verification of optimality to a test of the controllability and observability for a novel, specially constructed transfer function, yielding a precise and computationally tractable certificate. Furthermore, our condition provides a rigorous explanation for why traditional parameterizations can lead to suboptimal stationary points. Second, we elevate the controller parameter space from conventional finite-dimensional settings to the infinite-dimensional space and develop a gradient-based algorithm in this setting, for which we provide a theoretical analysis establishing global convergence. Finally, representative numerical experiments validate the theoretical findings and demonstrate the practical viability of the proposed approach. Additionally, the appendix section explores a data-driven extension to the model-free setting, where we outline a parameter estimation scheme and demonstrate its practical viability through numerical simulation.

Paper Structure

This paper contains 21 sections, 11 theorems, 147 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. The optimal condition of problem \ref{['op_lqg']}
  4. The properties of problem \ref{['op_lqg']}
  5. Optimal condition analysis
  6. The difference of optimality conditions of LQG and LQR
  7. Optimization method based on the optimality condition
  8. Optimization over the Space $\mathcal{RH}_\infty$
  9. The Gradient-Based Method and Convergence Analysis
  10. Numerical Experiments
  11. Example 1: Escaping a Suboptimal Stationary Point
  12. Example 2: Validation of Data-Driven Estimation Methods
  13. Experimental Setup
  14. Validation of $\mathbf{M}_{22}^{\mathcal{K}_0}(s)$ Estimation
  15. Validation of $\mathbf{S}_0(s)$ Estimation
  16. ...and 6 more sections

Key Result

Lemma 3.1

zheng2024benignnonconvexlandscapesoptimal If $q=n$ and $\mathcal{K}^*$ is a stationary point of problem op_lqg that satisfies or then $\mathcal{K}^*$ is an optimal solution to problem op_lqg.

Figures (3)

  • Figure 1: Convergence comparison between vanilla policy gradient and Algorithm \ref{['alg:Q_update']} when initialized (a) near and (b) exactly at a sub-optimal stationary point.
  • Figure 2: Convergence of the $\mathcal{H}_2$ estimation error for the non-zero elements of the sensitivity matrix $\mathbf{S}_0(s)$: comparison between Laguerre basis expansion and reduced-order model (both as functions of Laguerre basis order).
  • Figure 3: Qualitative convergence of the zeroth-order gradient estimate as the number of Monte Carlo samples ($m$) increases.

Theorems & Definitions (26)

  • Lemma 3.1
  • Lemma 3.2
  • proof
  • Lemma 3.3
  • Theorem 3.4
  • proof
  • Remark 3.5
  • Corollary 3.6
  • proof
  • Theorem 4.1
  • ...and 16 more