Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mixed memories in Hopfield networks

Véronique Gayrard

TL;DR

This work analyzes spurious memories in Hopfield networks by constructing and characterizing mixed memories, a broad class of local minima formed as deterministic mixtures of a finite subset of stored patterns. It introduces a Rademacher-system framework to convert random pattern equations into tractable deterministic equations, enabling explicit construction of $n$-mixed memories with odd $n$ and detailed overlap properties. The authors derive rigorous, model-specific growth conditions on the pattern count $M(N)$ under which these mixed memories become exact fixed points of retrieval dynamics for classical, dense, and modern Hopfield models, thereby illuminating when spurious memories robustly emerge. The results unify and extend prior numerical observations, provide explicit admissible mixture sets ${\mathcal M}_{n,F}$, and offer a conjecture that all local minima are captured by these mixed memories in the valid regime, with clear implications for memory capacity and energy landscape structure in analogue-to-binary neuron mappings.

Abstract

We consider the class of Hopfield models of associative memory with activation function $F$ and state space $\{-1,1\}^N$, where each vertex of the cube describes a configuration of $N$ binary neurons. $M$ randomly chosen configurations, called patterns, are stored using an energy function designed to make them local minima. If they are, which is known to depend on how $M$ scales with $N$, then they can be retrieved using a dynamics that decreases the energy. However, storing the patterns in the energy function also creates unintended local minima, and thus false memories. Although this has been known since the earliest work on the subject, it has only been supported by numerical simulations and non-rigorous calculations, except in elementary cases. Our results are twofold. For a generic function $F$, we explicitly construct a set of configurations, called mixed memories, whose properties are intended to characterise the local minima of the energy function. For three prominent models, namely the classical, the dense and the modern Hopfield models, obtained for quadratic, polynomial and exponential functions $F$ respectively, we give conditions on the growth rate of $M$ which guarantee that, as $N$ diverges, mixed memories are fixed points of the retrieval dynamics and thus exact minima of the energy. We conjecture that in this regime, all local minima are mixed memories.

Mixed memories in Hopfield networks

TL;DR

This work analyzes spurious memories in Hopfield networks by constructing and characterizing mixed memories, a broad class of local minima formed as deterministic mixtures of a finite subset of stored patterns. It introduces a Rademacher-system framework to convert random pattern equations into tractable deterministic equations, enabling explicit construction of -mixed memories with odd and detailed overlap properties. The authors derive rigorous, model-specific growth conditions on the pattern count under which these mixed memories become exact fixed points of retrieval dynamics for classical, dense, and modern Hopfield models, thereby illuminating when spurious memories robustly emerge. The results unify and extend prior numerical observations, provide explicit admissible mixture sets , and offer a conjecture that all local minima are captured by these mixed memories in the valid regime, with clear implications for memory capacity and energy landscape structure in analogue-to-binary neuron mappings.

Abstract

We consider the class of Hopfield models of associative memory with activation function and state space , where each vertex of the cube describes a configuration of binary neurons. randomly chosen configurations, called patterns, are stored using an energy function designed to make them local minima. If they are, which is known to depend on how scales with , then they can be retrieved using a dynamics that decreases the energy. However, storing the patterns in the energy function also creates unintended local minima, and thus false memories. Although this has been known since the earliest work on the subject, it has only been supported by numerical simulations and non-rigorous calculations, except in elementary cases. Our results are twofold. For a generic function , we explicitly construct a set of configurations, called mixed memories, whose properties are intended to characterise the local minima of the energy function. For three prominent models, namely the classical, the dense and the modern Hopfield models, obtained for quadratic, polynomial and exponential functions respectively, we give conditions on the growth rate of which guarantee that, as diverges, mixed memories are fixed points of the retrieval dynamics and thus exact minima of the energy. We conjecture that in this regime, all local minima are mixed memories.

Paper Structure

This paper contains 20 sections, 26 theorems, 345 equations.

Table of Contents

  1. Introduction
  2. Hopfield models
  3. Main results
  4. Mixed memories
  5. Mixed memories are exact local minima
  6. Preparatory tools
  7. The Rademacher system
  8. Solutions of the mixed-memory equations in the Rademacher system
  9. Proofs of the results of Section \ref{['S2.2']}
  10. First step.
  11. General construction step.
  12. Last step.
  13. Reduction to the Rademacher system
  14. Conditions for exact retrieval of mixed solutions
  15. Proof of Theorem \ref{['1.theo1']}
  16. ...and 5 more sections

Key Result

Theorem 1.2

Let $F$ be a smooth function whose derivative $F'$ satisfies $F'(x)>0$, for all $x>0$. For any odd $n\in{\Bbb N}$, if $m\in{\mathcal{M}}_{n,F}$ then $\xi^{(N)}(m)$ is an $n$-mixed memory of type $F$.

Theorems & Definitions (56)

  • Definition 1.1: Mixed memories of type $F$
  • Remark
  • Theorem 1.2: Mixed memories of type $F$
  • Proposition 1.3: Bounds on the number of mixed memories
  • Conjecture 1.4
  • Theorem 1.5: Classical Hopfield network
  • Theorem 1.6: Dense Hopfield network
  • Theorem 1.7: Modern Hopfield network
  • Remark
  • Lemma 1.8
  • ...and 46 more