How do Transformers Learn Implicit Reasoning?

TL;DR

This work investigates how Transformer models develop implicit multi-hop reasoning in a controlled symbolic environment, disentangling genuine internal reasoning from memorization. By training a decoder-only Transformer on atomic triples and multi-hop queries, and by employing cross-query semantic patching and a cosine-based representational lens, it uncovers a three-stage developmental trajectory: memorization, in-distribution generalization, and cross-distribution generalization. Key findings include that second-hop generalization requires query-level exposure to exact compositional structures, ID triples accelerate but are not strictly necessary for ID generalization, and successful cross-distribution reasoning correlates with cosine-space clustering of intermediate representations anchored by ID supervision. The work provides mechanistic, interpretable diagnostics linking training dynamics to internal geometry, offering pathways to enhance transparency of implicit reasoning in LLMs and informing data-design choices to improve generalization across distributions.

Abstract

Recent work suggests that large language models (LLMs) can perform multi-hop reasoning implicitly -- producing correct answers without explicitly verbalizing intermediate steps -- but the underlying mechanisms remain poorly understood. In this paper, we study how such implicit reasoning emerges by training transformers from scratch in a controlled symbolic environment. Our analysis reveals a three-stage developmental trajectory: early memorization, followed by in-distribution generalization, and eventually cross-distribution generalization. We find that training with atomic triples is not necessary but accelerates learning, and that second-hop generalization relies on query-level exposure to specific compositional structures. To interpret these behaviors, we introduce two diagnostic tools: cross-query semantic patching, which identifies semantically reusable intermediate representations, and a cosine-based representational lens, which reveals that successful reasoning correlates with the cosine-base clustering in hidden space. This clustering phenomenon in turn provides a coherent explanation for the behavioral dynamics observed across training, linking representational structure to reasoning capability. These findings provide new insights into the interpretability of implicit multi-hop reasoning in LLMs, helping to clarify how complex reasoning processes unfold internally and offering pathways to enhance the transparency of such models.

Figures (15)

• Figure 1: We track the evolution of bridge entity representations across training. In the memory phase (left), intermediate entities are explicitly decodable but geometrically scattered. In the implicit reasoning phase (right), representations converge in cosine space, supporting structured multi-hop reasoning through early-to-late layer transitions. Queries that share an bridge entity are composed for cosine-based representation analysis.
• Figure 2: Training dynamics under the base configuration, revealing three distinct phases. Accuracy curves track model performance on different query types, while dashed lines plot the ID Cohesion and OOD Alignment scores (Section \ref{['sec:cosine']}).
• Figure 3: Training trajectories under two configurations used in Section \ref{['sec:ID-triples-necessity']}. (a) Only Train-II queries are used, with no exposure to atomic ID triples. (b) Both Train-II queries and atomic ID triples are included in training.
• Figure 4: Average patching success rate across layers and token positions.
• Figure 5: Cosine-space projection of a random intermediate entity across three training phases.