Dynamical stability for dense patterns in discrete attractor neural networks

TL;DR

This work develops a comprehensive theory of dynamical stability for dense, graded-activation memory patterns in auto-associative networks. By combining replica mean-field analysis with a detailed Jacobian spectrum study, it identifies a distinct stability phase transition at a load $\alpha_S$ below the storage-capacity threshold $\alpha_C$, and shows stability is governed by a bulk spectrum and two outliers tied to average and memory-pattern structure. The analysis reveals that near-linear activation with a negative threshold and finite noise maximizes stable recall, with sparse-like patterns further enhancing stability under certain statistics. These results provide concrete design principles for biologically plausible memory networks and offer a framework applicable to other optimized high-dimensional systems with pseudo-inverse solutions.

Abstract

Neural networks storing multiple discrete attractors are canonical models of biological memory. Previously, the dynamical stability of such networks could only be guaranteed under highly restrictive conditions. Here, we derive a theory of the local stability of discrete fixed points in a broad class of networks with graded neural activities and in the presence of noise. By directly analyzing the bulk and the outliers of the Jacobian spectrum, we show that all fixed points are stable below a critical load that is distinct from the classical \textit{critical capacity} and depends on the statistics of neural activities in the fixed points as well as the single-neuron activation function. Our analysis highlights the computational benefits of threshold-linear activation and sparse-like patterns.

Figures (8)

• Figure 1: Storing fixed-points. (a) Color code for combinations of I/O exponent (x-axis) and smoothness (y-axis, log-scale). The activation function for selected combinations is illustrated (insets, top). These combinations are used in b-e, with $\textrm{CV}=2, \theta=-2$. (b,c) The sum (b) and L2-norm (c) of the presynaptic weights of a neuron (i.e. a row of $\mathbf{W}$) as a function of memory load ($\alpha$) for different activation functions (colors as in a). Note the divergence of L2-norm as $\alpha\rightarrow1$, as predicted by theory. (d) Measures of the weights asymmetry (top) and dynamics non-normality (bottom) at different load values (x-axis), for different activation functions (colors as in a). (e) The fraction of patterns which are correctly stored (thick curves) or stable for recall (thin curves) as a function of memory load ($\alpha$) for different activation functions (colors as in a). (f) Probability density of log-normal distributed patterns at different values of CV (color coded). The inset shows normalized density on a log scale. (g-j) Same as b-e for different values of pattern CV (colors as in f) with $\sigma=1, n=1, \theta=-2$. Solid vs. dotted lines in b,c,e, and g,h,i show numerical vs. theoretical results (with only numerical results shown for stability e,j -- see subsequent figures for corresponding theoretical results). Numerical simulations optimized the weights $\mathbf{W}$ according to \ref{['eq:optW']} at $N=256$.
• Figure 2: Fixed-point stability. (a) Example spectrum - overlaid eigenvalues of $P$ Jacobians $\mathbf{J_\mu}$ of a single problem (parameters indicated by text). Annotations of the theoretical values $\lambda_\mathrm{bulk}$ (blue line), $\lambda_{\mathrm{ave}}$ (green line), and $\lambda_{\mathrm{mem}}$ (orange line). The corresponding empirical eigenvalues are marked with matching colors: largest eigenvalue of the bulk (blue dot), the eigenvalue associated with the Jacobian average (green dot), and the eigenvalue associate with the memory pattern (orange dot). Inset shows a near-linear relation between $r^\mu_i$ (x-axis) and $\phi(r^\mu_i)$ (y-axis), a prerequisite for an outlier $\lambda_\mathrm{mem}$. (b) Same as a, for six parameter sets; columns differ by the maximal theoretical value. (c-e) Probability densities at different CV values (color-coded) of the difference between the empirical and predicted value of $\lambda_\mathrm{bulk}$ (c), $\lambda_\mathrm{ave}$ (d), and $\lambda_\mathrm{mem}$ (e), calculated over different choice of the parameters discussed in \ref{['fig:capacity', 'fig:SM_capacity']}, in cases where $\left|\lambda_\mathrm{bulk}\right|<10$ (c), $\left|\lambda_\mathrm{ave}\right|<10$ (d), $\tau_\mathrm{mem}>0.95$ (e).
• Figure 3: Phase diagrams for the critical load for stability. (a-b) Numerical simulations results for the critical load for stability $\alpha_\mathrm{S}$ (grayscale map) using different choices of I/O exponent $n$, smoothness $\sigma$, and threshold $\theta$, and theoretical predictions for the zero-crossings of the different terms in \ref{['eq:alphaS']}, i.e. $\alpha_\mathrm{S}^\mathrm{bulk}$, $\lambda_\mathrm{ave}$, and $\lambda_\mathrm{mem}$ (blue, green, and orange curves, respectively). $\lambda_\mathrm{mem}$ is considered only in the domain where $\tau_\mathrm{mem}>0.95$. In (a), two parameters are varied (axes), while the third parameter (title) is kept fixed. In (b), the third parameter is optimized for each combination of the two parameters that are being varied. $\mathrm{CV}=4$ in all cases shown. (c) The maximal value of $\alpha_\mathrm{S}$ for each value of $n$ (x-axis) and $\mathrm{CV}$ (color coded). (Parameters $\sigma$ and $\theta$ are optimized for each combination of $n$ and $\mathrm{CV}$.) (d) The maximal value of $\alpha_\mathrm{S}$ for each value of $\sigma$ (x-axis) and $\mathrm{CV}$ (color coded as in c). (Parameters $n$ and $\theta$ are optimized for each combination of $n$ and $\mathrm{CV}$.)
• Figure S1: Capacity phase-transition for sparse patterns. (a) Probability density of sparse, log-normal distributed patterns at different variation levels (CV, color coded). A fraction $1-f$ of the density is exactly at 0 (arrow). The inset shows normalized density on a log scale. (b) Rectified power activation with threshold $\theta$ and different exponents $n$. (c,d) Weights matrix row sum (c) and L2-norm (d) at different sparseness levels (color coded; empirical - full line, theory - dotted line) and loads (x-axis). (e-f) The fraction of patterns that are correctly stored in simulations (full line) at different loads (x-axis), and different sparseness levels (e, color coded), or different values of $N$ (f, see legend), or different values of $n$ and $\textrm{CV}$ (f, insets). Theory's predictions on the critical load - dotted lines. (g) The relation between $f$ and $\alpha_\mathrm{C}$ in linear scale or log-log scale (inset).
• Figure S2: Stability phase-transition for sparse patterns. (a-b) The fraction of patterns which are stable for recall in simulations (full line) at different loads (x-axis), and different sparseness levels (a, color coded), or different values of $N$ (b, see legend). The empirically found critical load $\alpha_\mathrm{S}$ - dotted lines. (c-d) The optimal choice for threshold $\theta$ (c) and mean spectral abscissa $\lambda^*$ (the real part of the eigenvalue with the largest real part) (d) at different loads (x-axis), and different sparseness levels (color coded), for the same experiments as a. (e) The empirically found critical load for stability $\alpha_\mathrm{S}$ at different levels of variation (panels), different I/O exponents (x-axis), and levels of sparseness (color coded). Dashed lines connect load values below the minimal value identifiable by the experiments.