A Biologically Plausible Dense Associative Memory with Exponential Capacity

TL;DR

A novel associative memory network with a threshold nonlinearity that enables distributed representations is introduced, enabling high-capacity, robust, and scalable architectures consistent with biological constraints.

Abstract

Krotov and Hopfield (2021) proposed a biologically plausible two-layer associative memory network with memory storage capacity exponential in the number of visible neurons. However, the capacity was only linear in the number of hidden neurons. This limitation arose from the choice of nonlinearity between the visible and hidden units, which enforced winner-take-all dynamics in the hidden layer, thereby restricting each hidden unit to encode only a single memory. We overcome this limitation by introducing a novel associative memory network with a threshold nonlinearity that enables distributed representations. In contrast to winner-take-all dynamics, where each hidden neuron is tied to an entire memory, our network allows hidden neurons to encode basic components shared across many memories. Consequently, complex patterns are represented through combinations of hidden neurons. These representations reduce redundancy and allow many correlated memories to be stored compositionally. Thus, we achieve much higher capacity: exponential in the number of hidden units, provided the number of visible units is sufficiently large relative to the number of hidden units. Exponential capacity arises because all binary states of the hidden units can become stable memory patterns. Moreover, the distributed hidden representation, which has much lower dimensionality than the visible layer, preserves class-discriminative structure, supporting efficient nonlinear decoding. These results establish a new regime for associative memory, enabling high-capacity, robust, and scalable architectures consistent with biological constraints.

Figures (11)

• Figure 1: Capacity versus the number of hidden units, $N_h$, with $N_v = 100 N_h$ and $\tau_v = 20 \tau_h$. (a) Capacity for different thresholds, $\theta$. The highest storage capacity is achieved when the threshold is set to its optimal theoretical value , $\theta=0.5$. (b) The effect of noise in the visible layer ($\epsilon_i^v$ in Eq. \ref{['noise']}), shown for different noise variances, demonstrates the large basin of attraction of the fixed points.
• Figure 2: Examples of recall in a network with 50 hidden neurons that memorized 60,000 MNIST images. Hidden neurons are shown on the ring, and visible neurons are visualized as two-dimensional images. On the ring, black indicates high activity, and white indicates low activity. Highly correlated images of every digit, for instance, the digit 6 shown here, converge to unique but overlapping hidden representations.
• Figure 3: a) 25 (out of 50) columns of the learned weight matrix for MNIST images, which serve as basic memories, are shown as two-dimensional images. b) Correlation matrix of the basic patterns, which correspond to the hidden units. c) The network generalizes compositionally, associating unseen cues with interpretable fixed points.
• Figure 4: Examples of recall in a network with 500 hidden neurons that memorized 50,000 CIFAR-10 images. Hidden neurons are arranged on a ring (50 out of 500), while visible neurons are shown as two-dimensional images. On the ring, black indicates high activity, and white indicates low activity.
• Figure 5: a) 25 (out of 500) columns of the learned weight matrix for CIFAR-10 images, which serve as basic memories, are shown as images. b) Correlation matrix of the basic patterns, which correspond to the hidden units. c) The network generalizes compositionally, associating unseen cues with interpretable fixed points.