Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integral Quadratic Constraints for Repeated ReLU

Sahel Vahedi Noori, Bin Hu, Geir Dullerud, Peter Seiler

Abstract

This paper presents a new dynamic integral quadratic constraint (IQC) for the repeated Rectified Linear Unit (ReLU). These dynamic IQCs can be used to analyze stability and induced $\ell_2$-gain performance of discrete-time, recurrent neural networks (RNNs) with ReLU activation functions. These analysis conditions can be incorporated into learning-based controller synthesis methods, which currently rely on static IQCs. We show that our proposed dynamic IQCs for repeated ReLU form a superset of the dynamic IQCs for repeated, slope-restricted nonlinearities. We also prove that the $\ell_2$-gain bounds are nonincreasing with respect to the horizon used in the dynamic IQC filter. A numerical example using a simple (academic) RNN shows that our proposed IQCs lead to less conservative bounds than existing IQCs.

Integral Quadratic Constraints for Repeated ReLU

Abstract

This paper presents a new dynamic integral quadratic constraint (IQC) for the repeated Rectified Linear Unit (ReLU). These dynamic IQCs can be used to analyze stability and induced -gain performance of discrete-time, recurrent neural networks (RNNs) with ReLU activation functions. These analysis conditions can be incorporated into learning-based controller synthesis methods, which currently rely on static IQCs. We show that our proposed dynamic IQCs for repeated ReLU form a superset of the dynamic IQCs for repeated, slope-restricted nonlinearities. We also prove that the -gain bounds are nonincreasing with respect to the horizon used in the dynamic IQC filter. A numerical example using a simple (academic) RNN shows that our proposed IQCs lead to less conservative bounds than existing IQCs.
Paper Structure (13 sections, 6 theorems, 40 equations, 3 figures, 3 tables)

This paper contains 13 sections, 6 theorems, 40 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Notation
  3. Problem statement
  4. Preliminary Results
  5. Integral Quadratic Constraints (IQCs)
  6. Stability and Performance Condition
  7. Main Results
  8. IQCs for Repeated Slope Restricted Nonlinearities
  9. Integral Quadratic Constraints for ReLU
  10. Stability Condition
  11. Numerical examples
  12. Conclusions
  13. Acknowledgments

Key Result

Lemma 1

Consider the interconnection $F_U(G,\Delta_F)$ where $G$ is the LTI system eq:LTInom and $\Delta_F:\ell_2^m \to \ell_2^m$ is a static, memoryless nonlinearity that satisfies the time-domain IQC defined by $(\Psi,M)$. Assume the interconnection is well-posed. Then the interconnection $F_U(G,\Delta_F)

Figures (3)

  • Figure 1: Interconnection $F_U(G,\Delta_F)$ of a nominal discrete-time LTI system $G$ and a static, memoryless nonlinearity $\Delta_F$.
  • Figure 2: Augmented LTI system $\hat{G}$ mapping $(w,d)$ to $(r,e)$ based on the interconnection of the nominal system $G$ and IQC filter $\Psi$.
  • Figure 3: Time domain graphical interpretation for filter $\Psi_N$.

Theorems & Definitions (16)

  • Definition 1
  • Definition 2
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Lemma 4
  • proof
  • ...and 6 more