Dense Associative Memory for Pattern Recognition

TL;DR

This work introduces dense associative memories with higher-order energy functions to achieve capacities far beyond traditional Hopfield networks, enabling reliable pattern retrieval of many more memories than neurons. It establishes a duality between these memories and one-hidden-layer neural networks, with the visible-to-hidden activation corresponding to the derivative of the energy function, and identifies a feature-to-prototype transition as the nonlinearity degree $n$ is varied. The authors provide capacity scaling laws showing nonlinear gains, illustrate with the XOR task, and demonstrate MNIST classification using a one-step memory update, where higher-degree rectified activations can improve generalization and speed. The results suggest a principled way to design activation functions and representations that interpolate between feature-based and prototype-based recognition, with potential benefits for larger-scale deep learning architectures.

Abstract

A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward neural networks with one hidden layer and various activation functions, which transmit the activities of the visible neurons to the hidden layer. This family of activation functions includes logistics, rectified linear units, and rectified polynomials of higher degrees. The proposed duality makes it possible to apply energy-based intuition from associative memory to analyze computational properties of neural networks with unusual activation functions - the higher rectified polynomials which until now have not been used in deep learning. The utility of the dense memories is illustrated for two test cases: the logical gate XOR and the recognition of handwritten digits from the MNIST data set.

Figures (5)

• Figure 1: (A) The network has $N=28\times28=784$ visible neurons and $N_c=10$ classification neurons. The visible units are clamped to intensities of pixels (which is mapped on the segment $[-1, 1]$), while the classification neurons are initialized in the state $x_\alpha$ and then updated once to the state $c_\alpha$. (B) Behavior of the error on the test set as training progresses. Each curve corresponds to a different combination of hyperparameters from the optimal window, which was determined on the validation set. The arrows show the first time when the error falls below a $2\%$ threshold. All models have $K=2000$ memories (hidden units).
• Figure 2: We show 25 randomly selected memories (feature detectors) for four networks, which use rectified polynomials of degrees $n=2,3,20,30$ as the energy function. The magnitude of a memory element corresponding to each pixel is plotted in the location of that pixel, the color bar explains the color code. The histograms at the bottom are explained in the text. The error rates refer to the particular four samples used in this figure. RU stands for recognition unit.
• Figure 3: On the left a feedforward neural network with one layer of hidden neurons. The states of the visible units are transformed to the hidden neurons using a non-linear function $f$, the states of the hidden units are transformed to the output layer using a non-linear function $g$. On the right the model of dense associative memory with one step update (\ref{['update_rule_MNIST']}). The two models are equivalent.
• Figure 4: The histograms of overlaps for models with $n=2,3,4$ with power energy functions (left) and rectified polynomial energy functions (right). Each histogram has 10000 samples in it.
• Figure 5: Scaling behavior of the capacity vs. the number of neurons for $n=3$ with power and rectified polynomial energy functions. Solid curve is the theoretical result (\ref{['perfect_capacity']}).