Emergent Analogical Reasoning in Transformers

TL;DR

The paper tackles how Transformers realize abstract analogy by formalizing analogical reasoning as functor‑driven correspondences between categories and by designing synthetic tasks that jointly test compositional and analogical generalization. It reveals a discriminative two‑part mechanism: geometric alignment of relational embeddings across categories and a functor‑like operation within Transformer layers that maps $e_s$ to its counterpart $e_t$ via $e_t \,\approx\, e_s + f$, with Dirichlet Energy $E$ serving as a quantitative anchor of alignment. The emergence of analogical reasoning is shown to be highly sensitive to data properties and optimization choices and does not monotonically scale with model size, though similar mechanistic signatures appear in pretrained LLMs under in‑context learning. Collectively, the work provides a mechanistic grounding for analogy in neural networks, bridging category theory, synthetic reasoning tasks, and large‑scale language models to illuminate cross‑domain generalization beyond sequential compositionality.

Abstract

Analogy is a central faculty of human intelligence, enabling abstract patterns discovered in one domain to be applied to another. Despite its central role in cognition, the mechanisms by which Transformers acquire and implement analogical reasoning remain poorly understood. In this work, inspired by the notion of functors in category theory, we formalize analogical reasoning as the inference of correspondences between entities across categories. Based on this formulation, we introduce synthetic tasks that evaluate the emergence of analogical reasoning under controlled settings. We find that the emergence of analogical reasoning is highly sensitive to data characteristics, optimization choices, and model scale. Through mechanistic analysis, we show that analogical reasoning in Transformers decomposes into two key components: (1) geometric alignment of relational structure in the embedding space, and (2) the application of a functor within the Transformer. These mechanisms enable models to transfer relational structure from one category to another, realizing analogy. Finally, we quantify these effects and find that the same trends are observed in pretrained LLMs. In doing so, we move analogy from an abstract cognitive notion to a concrete, mechanistically grounded phenomenon in modern neural networks.

Figures (17)

• Figure 1: (A) Synthetic task for compositional and analogical reasoning. Compositional reasoning evaluates whether a model can combine facts observed in-distribution (ID) during training to infer novel combinations (out-of-distribution, OOD). Analogical reasoning assesses whether a mapping $f$ (functor) between distinct categories generalizes. Solving analogical reasoning requires capturing the underlying relational structure of each category from the ID facts. (B) Training dynamics of Transformer. When training a Transformer on this task, the model first fits on in-distribution data, then acquires compositional reasoning, and finally succeeds at analogical reasoning. (C) Mechanism of analogical reasoning. We analyze internal representations of the Transformer before and after the emergence of analogical reasoning. After acquiring the ability for analogical reasoning, the model develops a well-structured embedding space, which is quantitatively characterized by a decrease in Dirichlet Energy.
• Figure 2: Learning dynamics under varying data properties in training data, compositionaland analogicalreasoning. From left to right, we vary (i) the number of entities ($|\mathcal{E}|$), (ii) the number of relations ($|\mathcal{R}|$), (iii) the compositional OOD ratio, and (iv) the analogical OOD ratio. While training accuracy and compositional generalization improve smoothly across settings, analogical reasoning consistently emerges later and exhibits unique behavior.
• Figure 3: Effects of optimization (left) and model scaling (right) on compositional and analogical reasoning. Moderate weight decay accelerates analogical reasoning, while excessively strong decay prevents it. Larger batch sizes lead to faster acquisition. Compositional reasoning improves consistently with model size, whereas analogical reasoning exhibits non-monotonic scaling.
• Figure 4: Analogical reasoning decomposes into two components: (1) Structural alignment in the embedding, (2) Functor application in Transformer.
• Figure 5: PCA visualization of entity embeddings before ($0$ step) and after ($10^3$ step) the acquisition of analogical reasoning. Entity embeddings from category $\mathcal{E}_1$ and $\mathcal{E}_2$ are shown, with arrows indicating the functor.

Theorems & Definitions (2)

• Definition 2.1: Compositional reasoning
• Definition 2.2: Analogical reasoning