Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Natural Gradient Descent for Control

Ramin Esmzad, Farnaz Adib Yaghmaie, Hamidreza Modares

TL;DR

This work addresses direct trajectory control for stochastic LTI systems by reframing controller design as a natural gradient descent problem preconditioned by the state covariance. The authors derive a Fisher Information Matrix-based preconditioner, show how stationary covariance drives the design via LMIs that yield a $\lambda$-contractive closed-loop with $K=F Y^{-1}$, and establish stability with explicit step-size bounds. A key contribution is representing the closed-loop dynamics through a gradient-flow perspective, enabling explicit trajectory shaping that is more interpretable than traditional LQR tuning. Simulation on a Quanser rotary pendulum demonstrates predictable, trajectory-focused behavior compared to standard LQR, highlighting the framework's potential for transparent low-level control and future extensions to nonlinear, time-varying, RL-integrated, and MPC-based settings.

Abstract

This paper bridges optimization and control, and presents a novel closed-loop control framework based on natural gradient descent, offering a trajectory-oriented alternative to traditional cost-function tuning. By leveraging the Fisher Information Matrix, we formulate a preconditioned gradient descent update that explicitly shapes system trajectories. We show that, in sharp contrast to traditional controllers, our approach provides flexibility to shape the system's low-level behavior. To this end, the proposed method parameterizes closed-loop dynamics in terms of stationary covariance and an unknown cost function, providing a geometric interpretation of control adjustments. We establish theoretical stability conditions. The simulation results on a rotary inverted pendulum benchmark highlight the advantages of natural gradient descent in trajectory shaping.

Natural Gradient Descent for Control

TL;DR

This work addresses direct trajectory control for stochastic LTI systems by reframing controller design as a natural gradient descent problem preconditioned by the state covariance. The authors derive a Fisher Information Matrix-based preconditioner, show how stationary covariance drives the design via LMIs that yield a -contractive closed-loop with , and establish stability with explicit step-size bounds. A key contribution is representing the closed-loop dynamics through a gradient-flow perspective, enabling explicit trajectory shaping that is more interpretable than traditional LQR tuning. Simulation on a Quanser rotary pendulum demonstrates predictable, trajectory-focused behavior compared to standard LQR, highlighting the framework's potential for transparent low-level control and future extensions to nonlinear, time-varying, RL-integrated, and MPC-based settings.

Abstract

This paper bridges optimization and control, and presents a novel closed-loop control framework based on natural gradient descent, offering a trajectory-oriented alternative to traditional cost-function tuning. By leveraging the Fisher Information Matrix, we formulate a preconditioned gradient descent update that explicitly shapes system trajectories. We show that, in sharp contrast to traditional controllers, our approach provides flexibility to shape the system's low-level behavior. To this end, the proposed method parameterizes closed-loop dynamics in terms of stationary covariance and an unknown cost function, providing a geometric interpretation of control adjustments. We establish theoretical stability conditions. The simulation results on a rotary inverted pendulum benchmark highlight the advantages of natural gradient descent in trajectory shaping.

Paper Structure

This paper contains 15 sections, 3 theorems, 32 equations, 3 figures.

Table of Contents

  1. Introduction
  2. System Model and Problem Description
  3. Basics of Gradients
  4. GD Algorithm
  5. Natural GD Algorithm
  6. From Basic GD to Natural GD in state-space
  7. Standard GD
  8. Preconditioned GD
  9. Natural GD
  10. Natural GD for Control
  11. Computing the FIM
  12. Controller Design Using Stationary Covariance
  13. Discussion
  14. Simulation Results
  15. Conclusion and Future Directions

Key Result

Theorem 1

Consider the system eq:syst and assume that $(A, B)$ is controllable. Assume that $\alpha>0$ is given and $0<\lambda<1$. Let $Y,\: F,\: \Sigma, \: M$ denote a feasible solution to the following linear problem Then, the preconditioned natural GD control in eq:ngd_ss with $V(x_k)=\mathbb{E}_{x_k \sim \mathcal{N}(\mu_k, \Sigma_k)}\left[x_k^{\top}Y^{-1} x_k\right]$ and $G(\mu_k)=\Sigma^{-1}$ makes th

Figures (3)

  • Figure 1: Quanser's Qube-Servo 2 platform
  • Figure 2: The evolution of the state variables (top four plots) and the control input (the fifth plot) using the natural GD design for various $\alpha$.
  • Figure 3: The evolution of the state variables (top four plots) and the control input (the fifth plot) using LQR approach with various $Q$ and $R$.

Theorems & Definitions (4)

  • Theorem 1
  • Theorem 2: Range of $\alpha$ for closed-Loop stability
  • Remark 1
  • Corollary 1