Dreaming improves memorization in a Hopfield model with bounded synaptic strength

TL;DR

It is shown that while clipping still removes catastrophic forgetting, alternating learning and dreaming phases improves the memorization capacity and makes the search for optimal performance more realistic from an evolutionary perspective.

Abstract

The Hopfield model provides a paradigmatic framework for associative memory. Its classical implementation, based on the Hebbian learning rule, suffers from catastrophic forgetting: when one attempts storing too many patterns, the network fails to retrieve any of them. Yet, the Hebbian rule does not take into account that synaptic strength is bounded. Introducing this biologically plausible modification, known as "clipping", eliminates catastrophic forgetting; the model is now able to retrieve the most recently seen memories, eliminating older ones. Yet, its memorization capacity is much reduced with respect to the unclipped case. Here, we investigate the effects of adding a "dreaming" phase on the capacity of a clipped Hopfield model. Following a proposal by Hopfield, Feinstein and Palmer, we assume that during the dreaming phase, the model generates random patterns that are then "unlearned". We show that while clipping still removes catastrophic forgetting, alternating learning and dreaming phases improves the memorization capacity and makes the search for optimal performance more realistic from an evolutionary perspective.

Figures (10)

• Figure 1: Pseudo-code of the Learning and Dreaming algorithm.
• Figure 1: Performance of standard (circles) and clipped (squares) Hebb rule as a function of the load. The dashed vertical line highlights the critical load $\alpha_c$ for the standard Hebb rule when $N\rightarrow\infty$. The results are averaged over 50 independent realizations of random memories.
• Figure 2: Recognition rate $\rho$ as a function of the dreaming steps for the standard (a) and clipped (b) dreaming algorithm, for different values of the load $\alpha$. In the standard case the maximum rate goes to zero for high values of $\alpha$ (catastrophic forgetting) while in the clipped case it goes to a constant, non-zero values, since the system remembers a fixed amount of the most recent patterns. Plots obtained with $N=200$ and averaged over 50 realizations of random memories.
• Figure 3: (a) Best performance achievable for standard (circles) and clipped (squares) dreaming algorithm as a function of the load. The results are obtained with $\tau_l =1$, $\tau_d=100$, and averaged over 50 independent realizations of random memories. (b) Scaling of the best possible $\rho$ with the number of neurons for the Hebb rule (blue) and the dreaming algorithm (red). The dashed lines are fits obtained with $f(N) = \rho_{\infty} + c_1/N + c_2/N^2$, where $\rho_{\infty}= 0.043$, $c_1 = 2.45$, $c_2=-106.65$ for the Hebb rule and $\rho_{\infty}= 0.139$, $c_1 = 3.64$, $c_2=-217.24$ for the dreaming algorithm.
• Figure 4: Performance of unclipped learning and dreaming algorithm for $N=200$, $\alpha=0.8$, and $\tau_l=\tau_d=10$. In (a) we report an average over the last 50 points of the $\rho(t)$ curve for a wide range of learning and dreaming steps. Each point is averaged over 50 realizations. In (b) we plot $\rho$ as a function of the epochs for different ratios of $L$ and $D$, highlighted with squares of the same color in the top panel. Each curve is averaged over 10 realizations.