Large Reasoning Models Are (Not Yet) Multilingual Latent Reasoners

TL;DR

This work investigates multilingual latent reasoning in large reasoning models (LRMs) across 11 languages using a truncation-based protocol to observe when correct answers emerge from partial reasoning traces on MGSM and Multilingual AIME. It introduces aggregate metrics—Area Under the Truncation Curve (AUTC), Area Under Gold-in-Trace Curve (AUGC), and Latent Reasoning Score (LRS)—and employs the logit lens and hidden-state cosine similarity to compare cross-language latent dynamics. Findings reveal that latent reasoning exists across languages but is stronger for resource-rich languages and weaker on harder benchmarks, with internal reasoning trajectories largely aligning with an English-centered pathway. Complementary perturbation analyses show partial memorization but confirm that latent reasoning persists, especially in higher-resource settings, and larger models improve but do not eliminate cross-language gaps. These insights have implications for multilingual reasoning data design, training, and alignment of LRMs across diverse languages.

Abstract

Large reasoning models (LRMs) achieve strong performance on mathematical reasoning tasks, often attributed to their capability to generate explicit chain-of-thought (CoT) explanations. However, recent work shows that LRMs often arrive at the correct answer before completing these textual reasoning steps, indicating the presence of latent reasoning -- internal, non-verbal computation encoded in hidden states. While this phenomenon has been explored in English, its multilingual behavior remains largely unknown. In this paper, we conduct a systematic investigation of multilingual latent reasoning in LRMs across 11 languages. Using a truncation-based strategy, we examine how the correct answer emerges as the model is given only partial reasoning traces, allowing us to measure stepwise latent prediction formation. Our results reveal clear evidence of multilingual latent reasoning, though unevenly: strong in resource-rich languages, weaker in low-resource ones, and broadly less observable on harder benchmarks. To understand whether these differences reflect distinct internal mechanisms, we further perform representational analyses. Despite surface-level disparities, we find that the internal evolution of predictions is highly consistent across languages and broadly aligns with English -- a pattern suggesting an English-centered latent reasoning pathway.

Figures (27)

• Figure 1: Pass@$k$ accuracy ($k=1,5,10$) and gold-in-trace rate under reasoning-trace truncation for R1-Qwen-32B. High accuracy with a low gold-in-trace rate indicates latent reasoning. The model shows strong evidence of latent reasoning in high-resource languages (e.g., English) on MGSM, but it is less detectable on Multilingual AIME.
• Figure 2: Causal decomposition of newly correct predictions across truncation intervals. Each bar partitions gains into three cases: (i) the gold answer is first articulated in the newly added reasoning steps, (ii) it was already articulated in earlier steps, or (iii) it has not yet appeared in the visible truncated trace. On MGSM, performance improvements at early and intermediate truncation ratios are dominated by case (iii), indicating that many gains arise from latent reasoning.
• Figure 3: Layer-wise rank of the gold answer obtained via logit lens across languages on MGSM (left three panels) and Multilingual AIME (right three panels). Rank trajectories exhibit highly similar trends across languages, suggesting that latent reasoning progresses through comparable layer-wise transformations regardless of language.
• Figure 4: Aggregated cosine similarity between hidden states in each language and English (reference), averaged over both reasoning steps and layers, for R1-Qwen-32B. High-resource languages show consistently higher similarity to English, suggesting convergence toward an English-centered latent reasoning pathway.
• Figure 5: Comparison of cosine similarity with English versus average similarity with other languages, shown separately for correctly and incorrectly solved examples. High-resource languages show stronger alignment with English, whereas low- and mid-resource languages show weaker or correctness-dependent alignment.