Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contracting Neural Networks: Sharp LMI Conditions with Applications to Integral Control and Deep Learning

Anand Gokhale, Anton V. Proskurnikov, Yu Kawano, Francesco Bullo

Abstract

This paper studies contractivity of firing-rate and Hopfield recurrent neural networks. We derive sharp LMI conditions on the synaptic matrices that characterize contractivity of both architectures, for activation functions that are either non-expansive or monotone non-expansive, in both continuous and discrete time. We establish structural relationships among these conditions, including connections to Schur diagonal stability and the recovery of optimal contraction rates for symmetric synaptic matrices. We demonstrate the utility of these results through two applications. First, we develop an LMI-based design procedure for low-gain integral controllers enabling reference tracking in contracting firing rate networks. Second, we provide an exact parameterization of weight matrices that guarantee contraction and use it to improve the expressivity of Implicit Neural Networks, achieving competitive performance on image classification benchmarks with fewer parameters.

Contracting Neural Networks: Sharp LMI Conditions with Applications to Integral Control and Deep Learning

Abstract

This paper studies contractivity of firing-rate and Hopfield recurrent neural networks. We derive sharp LMI conditions on the synaptic matrices that characterize contractivity of both architectures, for activation functions that are either non-expansive or monotone non-expansive, in both continuous and discrete time. We establish structural relationships among these conditions, including connections to Schur diagonal stability and the recovery of optimal contraction rates for symmetric synaptic matrices. We demonstrate the utility of these results through two applications. First, we develop an LMI-based design procedure for low-gain integral controllers enabling reference tracking in contracting firing rate networks. Second, we provide an exact parameterization of weight matrices that guarantee contraction and use it to improve the expressivity of Implicit Neural Networks, achieving competitive performance on image classification benchmarks with fewer parameters.

Paper Structure

This paper contains 17 sections, 13 theorems, 15 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Relevant Literature
  4. Contributions
  5. Background
  6. Notation
  7. Contraction theory
  8. Matrix inequalities for system analysis
  9. Contractivity of Lur'e systems
  10. Contraction of firing rate and Hopfield Neural networks
  11. Structural relationships of contractivity conditions
  12. Necessary versus sufficient conditions for contractivity
  13. Applications to integral control and machine learning
  14. Parameterization
  15. LMI conditions for integral control
  16. ...and 2 more sections

Key Result

Lemma 1

Let $Q \in \mathbb{R}^{n \times n}$ be a positive diagonal matrix. If an elementwise nonlinearity $\Psi \colon \mathbb{R}^n \rightarrow \mathbb{R}^n$ is slope-restricted in $[k_1, k_2]$ (where $k_1 \leq k_2$), it admits the multiplier matrix: $\blacktriangleleft$$\blacktriangleleft$

Figures (2)

  • Figure 3: A summary of relationships for the contractivity conditions from Table \ref{['tab:lmi_conditions']}. The sets $\mathcal{W}(\cdot, \cdot, \cdot)$ are described in Theorem \ref{['thm:structural_relationships']}. The discrete time CONE condition restricts the weight matrices the most, whereas the continuous time MONE condition enables maximum expressivity.
  • Figure 4: We perform system identification on a standard two-tank system, using an FRNN parameterized via Theorem \ref{['thm:parameterization']}. We design the gain via a low-gain controller as described in Theorem \ref{['thm:dc_gain']}.

Theorems & Definitions (17)

  • Definition 1: Incremental multiplier matrix
  • Lemma 1
  • Definition 2: CONE and MONE nonlinearities
  • Corollary 2
  • Lemma 3: Absolute contractivity of Lur'e systems
  • Remark 4
  • Theorem 5: Contractivity of FRNN and HNN
  • Lemma 6: Schur diagonal stability
  • Theorem 7: Reductions and duality of matrix inequalities
  • Corollary 8
  • ...and 7 more