Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stability and Performance Analysis of Discrete-Time ReLU Recurrent Neural Networks

Sahel Vahedi Noori, Bin Hu, Geir Dullerud, Peter Seiler

TL;DR

This work addresses stability and $\ell_2$-gain analysis for discrete-time RNNs with ReLU activations in feedback around an LTI plant. It fuses Lyapunov/dissipativity theory with Quadratic Constraints tailored to repeated ReLUs and employs a lifted $N$-step representation to derive LMI-based conditions that certify internal stability and bound the induced $\ell_2$-gain. A key theoretical contribution is showing that a general QC class for repeated ReLUs suffices, and that positive homogeneity does not expand this class beyond the most general QC, while lifting can reduce conservatism. Numerical examples on Lurye systems demonstrate horizon-dependent improvements in stability margins and tighter $\ell_2$-gain bounds when using ReLU-specific QCs versus generic slope-restricted QCs. The results offer a principled, verifiable framework for analyzing ReLU RNNs in control loops, with implications for inner-loop controller design and robustness analysis.

Abstract

This paper presents sufficient conditions for the stability and $\ell_2$-gain performance of recurrent neural networks (RNNs) with ReLU activation functions. These conditions are derived by combining Lyapunov/dissipativity theory with Quadratic Constraints (QCs) satisfied by repeated ReLUs. We write a general class of QCs for repeated RELUs using known properties for the scalar ReLU. Our stability and performance condition uses these QCs along with a "lifted" representation for the ReLU RNN. We show that the positive homogeneity property satisfied by a scalar ReLU does not expand the class of QCs for the repeated ReLU. We present examples to demonstrate the stability / performance condition and study the effect of the lifting horizon.

Stability and Performance Analysis of Discrete-Time ReLU Recurrent Neural Networks

TL;DR

This work addresses stability and -gain analysis for discrete-time RNNs with ReLU activations in feedback around an LTI plant. It fuses Lyapunov/dissipativity theory with Quadratic Constraints tailored to repeated ReLUs and employs a lifted -step representation to derive LMI-based conditions that certify internal stability and bound the induced -gain. A key theoretical contribution is showing that a general QC class for repeated ReLUs suffices, and that positive homogeneity does not expand this class beyond the most general QC, while lifting can reduce conservatism. Numerical examples on Lurye systems demonstrate horizon-dependent improvements in stability margins and tighter -gain bounds when using ReLU-specific QCs versus generic slope-restricted QCs. The results offer a principled, verifiable framework for analyzing ReLU RNNs in control loops, with implications for inner-loop controller design and robustness analysis.

Abstract

This paper presents sufficient conditions for the stability and -gain performance of recurrent neural networks (RNNs) with ReLU activation functions. These conditions are derived by combining Lyapunov/dissipativity theory with Quadratic Constraints (QCs) satisfied by repeated ReLUs. We write a general class of QCs for repeated RELUs using known properties for the scalar ReLU. Our stability and performance condition uses these QCs along with a "lifted" representation for the ReLU RNN. We show that the positive homogeneity property satisfied by a scalar ReLU does not expand the class of QCs for the repeated ReLU. We present examples to demonstrate the stability / performance condition and study the effect of the lifting horizon.
Paper Structure (12 sections, 6 theorems, 26 equations, 2 figures, 3 tables)

This paper contains 12 sections, 6 theorems, 26 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Notation
  3. Problem statement
  4. Main Results
  5. Quadratic Constraints for Slope-Restricted Functions
  6. Quadratic Constraints for ReLU
  7. Conditions for Stability and Performance
  8. Numerical examples
  9. Stability Analysis
  10. Gain Example
  11. Conclusions
  12. Acknowledgments

Key Result

Lemma 1

Let $\Phi:\mathbb{R}^{m} \to \mathbb{R}_{\ge 0}^m$ be a repeated nonlinearity with $\Phi(0)=0$ and slope restricted (elementwise) to $[0,1]$. If $Q_0 \in \mathbb{R}^{m \times m}$ is doubly hyperdominant then the following QC holds $\forall v\in\mathbb{R}^{m}$ and $w=\Phi(v)$:

Figures (2)

  • Figure 1: Left: Interconnection $F_U(G,\Phi)$ of a nominal discrete-time LTI system $G$ and repeated ReLU $\Phi$. Right: Graph of scalar ReLU $\phi$.
  • Figure 2: Lurye System

Theorems & Definitions (12)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • Lemma 5
  • proof
  • ...and 2 more