Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Complete Set of Quadratic Constraints for Repeated ReLU and Generalizations

Sahel Vahedi Noori, Bin Hu, Geir Dullerud, Peter Seiler

TL;DR

The paper addresses tightening stability and robustness analysis for networks with ReLU activations by deriving a complete class of quadratic constraints (QCs) for the repeated ReLU, characterized by a copositivity-based set $\mathcal{M}_c$ with $2^{n_v}$ conditions. It proves that only the repeated ReLU $\Phi$ and its flipped version $\Phi^{\text{flip}}$ satisfy all QCs in $\mathcal{M}_c$, and it further develops a complete incremental QC set $\mathcal{M}_c^{\text{inc}}$ along with a new incremental QC set $\mathcal{M}^{\text{inc}}_2$, enabling less conservative Lipschitz bounds and $\ell_2$-gain analyses via SDP relaxations. The framework extends to affine transformations and other piecewise linear activations (e.g., leaky ReLU, Householder, MaxMin) and is demonstrated through examples showing reduced conservatism compared to standard QC sets. This leads to practical SDP-based tools for stability and performance assessment of RNNs/NNs with ReLU activations and offers a unifying view of ReLU QCs with potential scalability improvements. Overall, the work significantly tightens the characterization of ReLU nonlinearities and provides actionable methods for rigorous analysis of Lipschitz bounds and stability in neural networks.

Abstract

This paper derives a complete set of quadratic constraints (QCs) for the repeated ReLU. The complete set of QCs is described by a collection of matrix copositivity conditions. We also show that only two functions satisfy all QCs in our complete set: the repeated ReLU and flipped ReLU. Thus our complete set of QCs bounds the repeated ReLU as tight as possible up to the sign invariance inherent in quadratic forms. We derive a similar complete set of incremental QCs for repeated ReLU, which can potentially lead to less conservative Lipschitz bounds for ReLU networks than the standard LipSDP approach. The basic constructions are also used to derive the complete sets of QCs for other piecewise linear activation functions such as leaky ReLU, MaxMin, and HouseHolder. Finally, we illustrate the use of the complete set of QCs to assess stability and performance for recurrent neural networks with ReLU activation functions. We rely on a standard copositivity relaxation to formulate the stability/performance condition as a semidefinite program. Simple examples are provided to illustrate that the complete sets of QCs and incremental QCs can yield less conservative bounds than existing sets.

A Complete Set of Quadratic Constraints for Repeated ReLU and Generalizations

TL;DR

The paper addresses tightening stability and robustness analysis for networks with ReLU activations by deriving a complete class of quadratic constraints (QCs) for the repeated ReLU, characterized by a copositivity-based set with conditions. It proves that only the repeated ReLU and its flipped version satisfy all QCs in , and it further develops a complete incremental QC set along with a new incremental QC set , enabling less conservative Lipschitz bounds and -gain analyses via SDP relaxations. The framework extends to affine transformations and other piecewise linear activations (e.g., leaky ReLU, Householder, MaxMin) and is demonstrated through examples showing reduced conservatism compared to standard QC sets. This leads to practical SDP-based tools for stability and performance assessment of RNNs/NNs with ReLU activations and offers a unifying view of ReLU QCs with potential scalability improvements. Overall, the work significantly tightens the characterization of ReLU nonlinearities and provides actionable methods for rigorous analysis of Lipschitz bounds and stability in neural networks.

Abstract

This paper derives a complete set of quadratic constraints (QCs) for the repeated ReLU. The complete set of QCs is described by a collection of matrix copositivity conditions. We also show that only two functions satisfy all QCs in our complete set: the repeated ReLU and flipped ReLU. Thus our complete set of QCs bounds the repeated ReLU as tight as possible up to the sign invariance inherent in quadratic forms. We derive a similar complete set of incremental QCs for repeated ReLU, which can potentially lead to less conservative Lipschitz bounds for ReLU networks than the standard LipSDP approach. The basic constructions are also used to derive the complete sets of QCs for other piecewise linear activation functions such as leaky ReLU, MaxMin, and HouseHolder. Finally, we illustrate the use of the complete set of QCs to assess stability and performance for recurrent neural networks with ReLU activation functions. We rely on a standard copositivity relaxation to formulate the stability/performance condition as a semidefinite program. Simple examples are provided to illustrate that the complete sets of QCs and incremental QCs can yield less conservative bounds than existing sets.
Paper Structure (25 sections, 19 theorems, 82 equations, 3 figures)

This paper contains 25 sections, 19 theorems, 82 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Notation
  4. QCs for ReLU
  5. Incremental QCs for ReLU
  6. Complete Sets of QCs and Incremental QCs for ReLU
  7. Complete Set of QCs for ReLU
  8. Functions Characterized by Complete Set of QCs
  9. Complete Set of Incremental QCs for ReLU
  10. New Incremental QCs for ReLU
  11. Related Extensions
  12. Affine Transformations
  13. Application to Householder and Max/Min Activations
  14. Stability Analysis With ReLU QCs
  15. Dynamic Systems With ReLU Activation Functions
  16. ...and 10 more sections

Key Result

Lemma 1

Define the following subsets of $\mathbb{S}^{2n_v}$: The ReLU $\Phi:\mathbb{R}^{n_v} \to \mathbb{R}_{\ge 0}^{n_v}$ satisfies the QCs defined by any matrices in $\mathcal{M}_{1}$ and $\mathcal{M}_{2}$, i.e. $\Phi\in QC(\mathcal{M}_{1})$ and $\Phi\in QC(\mathcal{M}_{2})$.

Figures (3)

  • Figure 1: Graph of scalar ReLU $\phi$.
  • Figure 2: Interconnection $F_U(G,\Phi)$ of a nominal discrete-time LTI system $G$ and ReLU $\Phi$.
  • Figure 3: Bound on induced $\ell_2$ gain for system $F_U(G,\Phi)$ vs. $\alpha$ using QCs defined by $\mathcal{M}_{c}$ and $\mathcal{M}_{12}$. We expect the interconnection to eventually become unstable as $\alpha$ increases which is consistent with both curves. The complete set $\mathcal{M}_{c}$ provides a less conservative (smaller) bound on the gain.

Theorems & Definitions (38)

  • Definition 1
  • Lemma 1
  • proof
  • Lemma 2
  • Definition 2
  • Theorem 1
  • proof
  • Lemma 3
  • proof
  • Corollary 1
  • ...and 28 more