Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unconditional stability of a recurrent neural circuit implementing divisive normalization

Shivang Rawat, David J. Heeger, Stefano Martiniani

TL;DR

This work links dynamic divisive normalization (DN) to the stability of “oscillatory recurrent gated neural integrator circuits” (ORGaNICs), a biologically plausible recurrent cortical circuit model that dynamically achieves DN and that has been shown to simulate a wide range of neurophysiological phenomena.

Abstract

Stability in recurrent neural models poses a significant challenge, particularly in developing biologically plausible neurodynamical models that can be seamlessly trained. Traditional cortical circuit models are notoriously difficult to train due to expansive nonlinearities in the dynamical system, leading to an optimization problem with nonlinear stability constraints that are difficult to impose. Conversely, recurrent neural networks (RNNs) excel in tasks involving sequential data but lack biological plausibility and interpretability. In this work, we address these challenges by linking dynamic divisive normalization (DN) to the stability of ORGaNICs, a biologically plausible recurrent cortical circuit model that dynamically achieves DN and that has been shown to simulate a wide range of neurophysiological phenomena. By using the indirect method of Lyapunov, we prove the remarkable property of unconditional local stability for an arbitrary-dimensional ORGaNICs circuit when the recurrent weight matrix is the identity. We thus connect ORGaNICs to a system of coupled damped harmonic oscillators, which enables us to derive the circuit's energy function, providing a normative principle of what the circuit, and individual neurons, aim to accomplish. Further, for a generic recurrent weight matrix, we prove the stability of the 2D model and demonstrate empirically that stability holds in higher dimensions. Finally, we show that ORGaNICs can be trained by backpropagation through time without gradient clipping/scaling, thanks to its intrinsic stability property and adaptive time constants, which address the problems of exploding, vanishing, and oscillating gradients. By evaluating the model's performance on RNN benchmarks, we find that ORGaNICs outperform alternative neurodynamical models on static image classification tasks and perform comparably to LSTMs on sequential tasks.

Unconditional stability of a recurrent neural circuit implementing divisive normalization

TL;DR

This work links dynamic divisive normalization (DN) to the stability of “oscillatory recurrent gated neural integrator circuits” (ORGaNICs), a biologically plausible recurrent cortical circuit model that dynamically achieves DN and that has been shown to simulate a wide range of neurophysiological phenomena.

Abstract

Stability in recurrent neural models poses a significant challenge, particularly in developing biologically plausible neurodynamical models that can be seamlessly trained. Traditional cortical circuit models are notoriously difficult to train due to expansive nonlinearities in the dynamical system, leading to an optimization problem with nonlinear stability constraints that are difficult to impose. Conversely, recurrent neural networks (RNNs) excel in tasks involving sequential data but lack biological plausibility and interpretability. In this work, we address these challenges by linking dynamic divisive normalization (DN) to the stability of ORGaNICs, a biologically plausible recurrent cortical circuit model that dynamically achieves DN and that has been shown to simulate a wide range of neurophysiological phenomena. By using the indirect method of Lyapunov, we prove the remarkable property of unconditional local stability for an arbitrary-dimensional ORGaNICs circuit when the recurrent weight matrix is the identity. We thus connect ORGaNICs to a system of coupled damped harmonic oscillators, which enables us to derive the circuit's energy function, providing a normative principle of what the circuit, and individual neurons, aim to accomplish. Further, for a generic recurrent weight matrix, we prove the stability of the 2D model and demonstrate empirically that stability holds in higher dimensions. Finally, we show that ORGaNICs can be trained by backpropagation through time without gradient clipping/scaling, thanks to its intrinsic stability property and adaptive time constants, which address the problems of exploding, vanishing, and oscillating gradients. By evaluating the model's performance on RNN benchmarks, we find that ORGaNICs outperform alternative neurodynamical models on static image classification tasks and perform comparably to LSTMs on sequential tasks.
Paper Structure (30 sections, 16 theorems, 128 equations, 7 figures, 7 tables, 2 algorithms)

This paper contains 30 sections, 16 theorems, 128 equations, 7 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model description
  4. Stability analysis of high-dimensional model with identity recurrent weights
  5. Stability analysis for arbitrary recurrent weights
  6. Experiments
  7. Static input classification task
  8. Time varying input
  9. Discussion
  10. Derivation of ORGaNICs
  11. Stability theorem
  12. Positive definite block matrices (Schur complement)
  13. Lyapunov stability criteria
  14. Analytical eigenvalue for the fully normalized circuit
  15. Convergent theorem
  16. ...and 15 more sections

Key Result

Theorem 4.1

For a system of linear differential equations with constant coefficients of the form, where $\mathbf{B} \in \mathbb{R}^{n \times n}$ and $\mathbf{K} \in \mathbb{R}^{n \times n}$ is a positive diagonal matrix (hence $\mathbf{K} \succ 0$), the dynamical system is globally asymptotically stable if $\mathbf{B}$ is Lyapunov diagonally stable.

Figures (7)

  • Figure 1: Phase portraits for 2D ORGaNICs with positive input drive. We plot the phase portraits of 2D ORGaNICs in the vicinity of the stable fixed points for contractive (a, d) and expansive (b, c, e, f) recurrence scalar $w_r$. A stable fixed point always exists, regardless of the parameter values. (a-c), The main model (Eq. \ref{['eq:main_madel_2D_eqns']}). (d-f), The rectified model (Eq. \ref{['eq:2d_rectified_organics_eqn']}). Red stars and black circles indicate stable and unstable fixed points, respectively. The parameters for all plots are: $b = 0.5$, $\tau_a = 2\,\text{ms}$, $\tau_y = 2\,\text{ms}$, $w = 1.0$, and $z = 1.0$. For (a) & (d), the parameters are $w_r = 0.5$, $b_0 = 0.5$, $\sigma = 0.1$; for (b) & (e), $w_r = 2.0$, $b_0 = 0.5$, $\sigma = 0.1$; and for (c) & (f), $w_r = 2.0$, $b_0 = 1.0$, $\sigma = 1.0$.
  • Figure 2: Phase portraits for 2D ORGaNICs with negative input drive. We plot the phase portraits of 2D ORGaNICs in the vicinity of the stable fixed point for contractive (a, d) and expansive (b, c, e, f) recurrence scalar $w_r$. A stable fixed point always exists, regardless of the parameter values. (a-c), The main model (Eq. \ref{['eq:main_madel_2D_eqns']}). (d-f), The rectified model (Eq. \ref{['eq:2d_rectified_organics_eqn']}). Red stars and black circles indicate stable and unstable fixed points, respectively. The parameters for all plots are: $b = 0.5$, $\tau_a = 2\,\text{ms}$, $\tau_y = 2\,\text{ms}$, $w = 1.0$, and $z = -1.0$. For (a) & (d), the parameters are $w_r = 0.5$, $b_0 = 0.5$, $\sigma = 0.1$; for (b) & (e), $w_r = 2.0$, $b_0 = 0.5$, $\sigma = 0.1$; and for (c) & (f), $w_r = 2.0$, $b_0 = 1.0$, $\sigma = 1.0$.
  • Figure 3: Phase portraits for 2D rectified ORGaNICs for different time constants. Red stars indicate stable fixed points. The parameters for all plots are: $w_r = 1.0$, $b_0 = 0.5$, $b = 0.5$, $\sigma = 0.1$, $w = 1.0$, and $z = 1.0$. For (a), the time constants are $\tau_a = 2\,\text{ms}$, $\tau_y = 2\,\text{ms}$; for (b), $\tau_a = 10\,\text{ms}$, $\tau_y = 2\,\text{ms}$; for (c), $\tau_a = 2\,\text{ms}$, $\tau_y = 10\,\text{ms}$.
  • Figure 4: Fast convergence of the iterative algorithm. The results are for 20-dimensional ORGaNICs (10 $\mathbf{y}$ and 10 $\mathbf{a}$ neurons) with random parameters and inputs with the additional constraint of the maximum singular value of $\mathbf{W}_r$ equal to 1 and $||\mathbf{z}|| < 1$. (a), Mean (with error bars representing 1-sigma S.D.) and maximum errors ($\epsilon$) as a function of number of iterations. $\epsilon$ is calculated as the norm of the difference between the true solution (found by simulation starting with random initialization) and the iteration solution. (b), An example of a randomly sampled $\mathbf{W}_r$. (c), Steady-state approximation as a function of iteration number. Different lines represent different neurons. (d), Overlap between the iteration solution (after 15 iterations) and the true solution.
  • Figure 5: Histogram for the eigenvalue with the largest real part. We train two-layer ORGaNICs ($\tau_a = \tau_y = 2\,\text{ms}$) with a static MNIST input where $\mathbf{W}_r$ is constrained to have a maximum singular value of 1. We plot the histogram of eigenvalues of the Jacobian matrix with the largest real part, for inputs from the test set. We find that all the eigenvalues of the Jacobian have negative real parts, implying asymptotic stability. (a), histogram for the first layer. (b), histogram for the second layer. Note that since this is implemented in a feedforward manner, this is a cascading system with no feedback, hence we can perform the stability analysis of the two layers independently.
  • ...and 2 more figures

Theorems & Definitions (24)

  • Theorem 4.1
  • Theorem 4.2
  • Theorem 4.3
  • Theorem 4.4
  • Theorem 4.5
  • Theorem 5.1
  • Theorem 5.2
  • Conjecture 5.3
  • Theorem B.1
  • proof
  • ...and 14 more