Rectified Lagrangian for Out-of-Distribution Detection in Modern Hopfield Networks

TL;DR

This work tackles the challenge of detecting out-of-distribution data in modern Hopfield networks by introducing Rectified Lagrangian (RecLag), a modified memory Lagrangian that creates an explicit OOD attractor while preserving the MHN framework. The authors prove theoretical guarantees including a trivial origin attractor for any interaction matrix and reduction to vanilla MHNs in the high memory-strength limit, and they train the interaction via probabilistic, SFNN-like mechanisms to estimate in-distribution densities. They further show that data with low probability under a derived joint distribution p_H fall into the OOD attractor, enabling density-based OOD detection, and demonstrate superior performance against energy-based baselines across nine image datasets. The approach offers a principled, post-hoc OOD detector for MHNs with tunable sensitivity via the inverse memory-strength parameter and potential for broader applicability to structured or hierarchical memory patterns.

Abstract

Modern Hopfield networks (MHNs) have recently gained significant attention in the field of artificial intelligence because they can store and retrieve a large set of patterns with an exponentially large memory capacity. A MHN is generally a dynamical system defined with Lagrangians of memory and feature neurons, where memories associated with in-distribution (ID) samples are represented by attractors in the feature space. One major problem in existing MHNs lies in managing out-of-distribution (OOD) samples because it was originally assumed that all samples are ID samples. To address this, we propose the rectified Lagrangian (RegLag), a new Lagrangian for memory neurons that explicitly incorporates an attractor for OOD samples in the dynamical system of MHNs. RecLag creates a trivial point attractor for any interaction matrix, enabling OOD detection by identifying samples that fall into this attractor as OOD. The interaction matrix is optimized so that the probability densities can be estimated to identify ID/OOD. We demonstrate the effectiveness of RecLag-based MHNs compared to energy-based OOD detection methods, including those using state-of-the-art Hopfield energies, across nine image datasets.

Figures (5)

• Figure 1: Visualization of energy landscapes. Existing methods identify samples with high Hopfield energy as OOD, though such samples fall into the attractors associated with ID samples. In contrast, our RecLag-based MHNs possess a dedicated attractor that specifically captures OOD samples.
• Figure 2: (a) MHN with an interaction matrix $\xi_{\mu i}$ between memory neurons $h_{\mu}$ and feature neurons $v_{i}$. (b) Energy distribution of a vanilla MHN using the Lagrangians in Eq. (\ref{['eq:lag_modern']}). (c) Energy distribution of a RecLag-based MNH. The point attractor in Theorem 1 created by RecLag is marked by the yellow star. (d) Probability density distribution in Eq. (\ref{['eq:reclagprob']}). Data samples with low probability density values fall into the created attractor.
• Figure 3: ROC curves (log-scale) and AUC$\uparrow$. Yellow background indicates that RecLag performed the best in terms of AUC.
• Figure 4: Comparison detection scores over time on LSUN-R. ResNet18 trained on CIFAR-10 was used.
• Figure 6: OOD samples incorrectly identified as ID samples. ID: CIFAR-100, OOD: TIN.

Theorems & Definitions (4)

• proof
• proof
• proof
• proof