Semantically-correlated memories in a dense associative model
Thomas F Burns
TL;DR
CDAM introduces a unified dense associative memory that jointly supports auto- and hetero-association on a graph-structured memory. Memory patterns are embedded in $oldsymbol\Xi \in \mathbb{R}^{n\times p}$ and recalled via a graph-augmented update $\sigma(t+1)=\sigma(t)+\eta\big([ \text{softmax}(\beta\,\sigma(t)\Xi)\,Q - \tfrac{1}{n}\tilde{\xi}^{\top}]-\sigma(t)\big)$ with $Q=a\Xi+h\Xi M^{T}$, where $M$ is the normalized adjacency of the memory graph. The analysis identifies four dynamical regimes—auto-, narrow hetero-, wide hetero-, and neutral quiescence—and shows that anti-Hebbian modulation (through $a$ and $h$) widens hetero-range, enables multi-scale community extraction, and stabilizes temporal sequences. The framework is demonstrated across synthetic graphs, real data, sparse video sequences, and finite-automata tasks, and is positioned to inform both neuroscience and Transformer interpretability by linking attention-like dynamics to energy-based hetero- and auto-associations.
Abstract
I introduce a novel associative memory model named Correlated Dense Associative Memory (CDAM), which integrates both auto- and hetero-association in a unified framework for continuous-valued memory patterns. Employing an arbitrary graph structure to semantically link memory patterns, CDAM is theoretically and numerically analysed, revealing four distinct dynamical modes: auto-association, narrow hetero-association, wide hetero-association, and neutral quiescence. Drawing inspiration from inhibitory modulation studies, I employ anti-Hebbian learning rules to control the range of hetero-association, extract multi-scale representations of community structures in graphs, and stabilise the recall of temporal sequences. Experimental demonstrations showcase CDAM's efficacy in handling real-world data, replicating a classical neuroscience experiment, performing image retrieval, and simulating arbitrary finite automata.