Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthesis of Stabilizing Recurrent Equilibrium Network Controllers

Neelay Junnarkar, He Yin, Fangda Gu, Murat Arcak, Peter Seiler

TL;DR

This work addresses stabilizing partially observed dynamical systems with sector-bounded nonlinearities while optimizing arbitrary reward structures. It introduces a recurrent equilibrium network (REN)–based controller, derives a Lyapunov condition, and develops a convex inner-approximation via loop transformation and LMIs to certify stability; a projection step enables gradient-based RL to synthesize controllers without sacrificing stability. The method extends from linear plants to nonlinear sector-bound components, using a two-stage projection when needed, and yields a practical synthesis workflow with exponential stability guarantees. Demonstrations on an inverted pendulum and a neural-network plant model show that the REN controller can achieve stable, high-performance control with smaller models than comparable RNN-based methods, including cases where the plant is NN-modeled.

Abstract

We propose a parameterization of a nonlinear dynamic controller based on the recurrent equilibrium network, a generalization of the recurrent neural network. We derive constraints on the parameterization under which the controller guarantees exponential stability of a partially observed dynamical system with sector bounded nonlinearities. Finally, we present a method to synthesize this controller using projected policy gradient methods to maximize a reward function with arbitrary structure. The projection step involves the solution of convex optimization problems. We demonstrate the proposed method with simulated examples of controlling nonlinear plants, including plants modeled with neural networks.

Synthesis of Stabilizing Recurrent Equilibrium Network Controllers

TL;DR

This work addresses stabilizing partially observed dynamical systems with sector-bounded nonlinearities while optimizing arbitrary reward structures. It introduces a recurrent equilibrium network (REN)–based controller, derives a Lyapunov condition, and develops a convex inner-approximation via loop transformation and LMIs to certify stability; a projection step enables gradient-based RL to synthesize controllers without sacrificing stability. The method extends from linear plants to nonlinear sector-bound components, using a two-stage projection when needed, and yields a practical synthesis workflow with exponential stability guarantees. Demonstrations on an inverted pendulum and a neural-network plant model show that the REN controller can achieve stable, high-performance control with smaller models than comparable RNN-based methods, including cases where the plant is NN-modeled.

Abstract

We propose a parameterization of a nonlinear dynamic controller based on the recurrent equilibrium network, a generalization of the recurrent neural network. We derive constraints on the parameterization under which the controller guarantees exponential stability of a partially observed dynamical system with sector bounded nonlinearities. Finally, we present a method to synthesize this controller using projected policy gradient methods to maximize a reward function with arbitrary structure. The projection step involves the solution of convex optimization problems. We demonstrate the proposed method with simulated examples of controlling nonlinear plants, including plants modeled with neural networks.
Paper Structure (18 sections, 5 theorems, 31 equations, 5 figures, 2 algorithms)

This paper contains 18 sections, 5 theorems, 31 equations, 5 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Notation
  4. Partially Observed Linear Systems
  5. Problem Formulation
  6. Controller Parameterization
  7. Quadratic Constraints for Activation Functions
  8. Loop Transformation
  9. Convex Stability Condition
  10. Stabilizing Reinforcement Learning Problem
  11. Partially Observed Linear Systems with Sector-Bounded Nonlinearity
  12. Problem Formulation
  13. Stability Condition
  14. Stabilizing Reinforcement Learning Problem
  15. Examples
  16. ...and 3 more sections

Key Result

Lemma 1

Let $\alpha_\phi, \beta_\phi \in \mathbb{R}^{n_\phi}$ be given with $\alpha_\phi \leq \beta_\phi$. Suppose that $\phi$ satisfies the sector bound $[\alpha_\phi, \beta_\phi]$ element-wise. For any $\Lambda \in \mathbb{D}_{+}^{n_\phi}$, and for all $v \in \mathbb{R}^{n_\phi}$ and $w = \phi(v)$, it hol where $A_\phi = \mathrm{diag}(\alpha_\phi)$, and $B_\phi = \mathrm{diag}(\beta_\phi)$.

Figures (5)

  • Figure 1: Controller $\pi_\theta$ as an interconnection of $P_\pi$ and $\phi$.
  • Figure 2: $\tanh \in$ sector $[0, 1]$.
  • Figure 3: Reward during training of our REN-based controller with $n_\xi = 2,\ n_\phi = 4$ and RNN controller from gu2021recurrent with $n_\xi = 2,\ n_\phi = 8$. The additional expressiveness of the REN-based controller and convexification procedure allows for slightly better performance with a smaller model. Note that stability is guaranteed at all points in training in both methods.
  • Figure 4: Phase portraits of the controller with stability guarantee of the true plant model vs the controller with stability guarantee of the learned plant model. Both controllers achieve convergence to the origin with all trajectories.
  • Figure 5: Reward of controller guaranteeing stability of learned plant model vs true plant model: The controller that only guarantees stability of the learned plant model achieves reward near that of the controller which has access to the true model parameters.

Theorems & Definitions (9)

  • Definition 1
  • Lemma 1
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Theorem 1
  • proof
  • Theorem 2