Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure

TL;DR

The paper addresses the Reversal Curse as a fundamental generalization gap in LLMs and reframes it as a binding problem in conceptual representations. It demonstrates that reversal can be learned at the abstract concept level using JEPA-based architectures and memory-enhanced recognition modules, revealing two core causes—conceptual inconsistency and entanglements—that hinder surface-level learning. By mitigating entanglements and enabling parametric memory, the approach achieves reversal with non-trivial generalization and enables parametric forward-chaining for large-scale arithmetic reasoning, outperforming non-parametric-memory LLMs on structured tasks. These findings highlight a path toward robust, memory-driven generalization while emphasizing the need for systematic binding mechanisms that go beyond manual scaffolding and surface-name representations.

Abstract

Despite their impressive capabilities, LLMs exhibit a basic generalization failure known as the Reversal Curse, where they struggle to learn reversible factual associations. Understanding why this occurs could help identify weaknesses in current models and advance their generalization and robustness. In this paper, we conjecture that the Reversal Curse in LLMs is a manifestation of the long-standing binding problem in cognitive science, neuroscience and AI. Specifically, we identify two primary causes of the Reversal Curse stemming from transformers' limitations in conceptual binding: the inconsistency and entanglements of concept representations. We perform a series of experiments that support these conjectures. Our exploration leads to a model design based on JEPA (Joint-Embedding Predictive Architecture) that for the first time breaks the Reversal Curse without side-stepping it with specialized data augmentation or non-causal masking, and moreover, generalization could be further improved by incorporating special memory layers that support disentangled concept representations. We demonstrate that the skill of reversal unlocks a new kind of memory integration that enables models to solve large-scale arithmetic reasoning problems via parametric forward-chaining, outperforming frontier LLMs based on non-parametric memory and prolonged explicit reasoning.

Figures (6)

• Figure 1: (a) We find that Transformers can learn reversal when inputs are represented and perceived at the abstract concept level. (b) Two conjectured causes of the Reversal Curse underlying surface-level predictions, both upon transformers' limitations in conceptual binding: 1) representational inconsistency when entities switch roles between perceived subjects and predicted objects (left); 2) representational entanglements cause interferences on the learning dynamics and impede generalization (right). Details in §\ref{['sec:ideal']} and §\ref{['sec:binding']}.
• Figure 2: Performance for JEPA with in-batch contrastive loss. Left: performance across varying depths of the recognition module (#Rec) and semantic module (#Sem). JEPA unlocks highly non-trivial generalization, but suffers from entanglements whose effects scale with model depth. Right: impact of multiplicity across different model configurations. Performance consistently and significantly degrades as multiplicity increases.
• Figure 3: Mitigating the effect of entanglements by increasing the model width (left) and using special memory layers for the recognition module (right). It can be seen that increasing the model width only brings incremental improvements, while memory layers,which eliminate entanglements by design, could boost generalization by a large margin.
• Figure 4: Left: illustration of the parametric variable binding enabled by models with reversal skills. Right: performance on the large-scale arithmetic reasoning task with various branching factors. "(P)": Parametric Memory. "(NP)": Non-Parametric Memory.
• Figure 5: Illustration of JEPA with in-batch contrastive learning.