Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Unstable Fixed Points in Modern Continuous Hopfield Networks

Hans-Peter Beise

Abstract

The recently introduced continuous Hopfield network (see Ramsauer et al.) exhibits large memorization capabilities, which manifest as attractive fixed points of its update rule -- a differentiable function consisting of two linear mappings composed with the scaled softmax function. The authors of the aforementioned work provide proofs for the existence and approximate position of such attractive fixed points. For the softmax function alone, the fixed point structure has been fully characterized in earlier work by P. Tiňo, from which it turns out that for sufficiently large scaling factors there are exponentially more unstable fixed points than attractive ones. In this work, we complement the findings of Ramsauer et al. by showing that, under natural geometric conditions on the vectors defining the continuous Hopfield network, unstable fixed points must occur, analogous to the findings of Tiňo. Our results show that, under these geometric conditions, continuous Hopfield networks necessarily admit additional unstable fixed points associated with higher-dimensional faces of the pattern polytope.

On Unstable Fixed Points in Modern Continuous Hopfield Networks

Abstract

The recently introduced continuous Hopfield network (see Ramsauer et al.) exhibits large memorization capabilities, which manifest as attractive fixed points of its update rule -- a differentiable function consisting of two linear mappings composed with the scaled softmax function. The authors of the aforementioned work provide proofs for the existence and approximate position of such attractive fixed points. For the softmax function alone, the fixed point structure has been fully characterized in earlier work by P. Tiňo, from which it turns out that for sufficiently large scaling factors there are exponentially more unstable fixed points than attractive ones. In this work, we complement the findings of Ramsauer et al. by showing that, under natural geometric conditions on the vectors defining the continuous Hopfield network, unstable fixed points must occur, analogous to the findings of Tiňo. Our results show that, under these geometric conditions, continuous Hopfield networks necessarily admit additional unstable fixed points associated with higher-dimensional faces of the pattern polytope.

Paper Structure

This paper contains 6 sections, 8 theorems, 73 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Main Result
  3. Mathematical Formulation and Proofs
  4. Numerical Considerations
  5. Conclusion
  6. The Fixed Points of the Softmax Mapping $\mathcal{S}_\beta$

Key Result

Theorem 2.1

Let $f:P\rightarrow \mathbb{R}^d$ be continuous on a convex, compact polytope $P\subset \mathbb{R}^d$ with non-empty interior. Let $H(F)$ denote the supporting hyperplane of a facet $F\in \mathcal{F}^{d-1}(P)$, where $H(F)=\{\boldsymbol{x}\in \mathbb{R}^d: \boldsymbol{n}_F^T \boldsymbol{x}=b_F\}$ fo Then $f$ has a fixed point in $P$.

Figures (3)

  • Figure 1: Dynamics of continuous Hopfield function (\ref{['networkFunHopf']}) for 2D data with $\beta=15$. Small dots depict the evolution of 3000 uniform random points in $[-1,1]^2$. Attractive fixed points, approximately equal to four vectors $\boldsymbol{w}_j$ defining $f$ are represented as large, bordered dots. The evolution after 0, 1, 2, 4, and 7 iterative applications of $f$ is shown from left to right. The colors correspond to the fixed point to which the points converge.
  • Figure 2: The triangle $A,\, B,\, C$, defines the polytope $P$, (solid) is centered at its arithmetic mean $\mathbf{b}_P$. The scaled polytope $P_\eta$ (dashed) contracts toward the arithmetic mean with factor $\eta$. The prism $(P_\eta)^\varepsilon = P_\eta + \varepsilon C(P^\perp)$ is shown with light gray faces.
  • Figure 3: Graphs and minima of $h_{n,k}$ (c.f. (\ref{['def_hk']}))

Theorems & Definitions (15)

  • Theorem 2.1: Poincaré--Miranda type fixed points
  • Definition 2.2
  • Theorem 2.3: Informal
  • Lemma 2.4
  • Remark 2.5
  • Theorem 3.1
  • Corollary 3.2
  • Remark 3.3
  • proof
  • Lemma 3.4
  • ...and 5 more