Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy

Jirui Qi, Shan Chen, Zidi Xiong, Raquel Fernández, Danielle S. Bitterman, Arianna Bisazza

TL;DR

This work examines multilingual reasoning in large reasoning models and reveals a robust trade-off between forcing models to think in a user's language and maintaining answer accuracy. It introduces the XReasoning benchmark to evaluate thinking-language alignment across 11 languages on math and science tasks, and shows that prompt hacking can dramatically increase language matching at a cost to accuracy, with the effect diminishing as model size grows. Targeted post-training with a small number of in-language examples improves alignment but further degrades performance, while model merging offers partial gains in matching and limited accuracy improvements. The study highlights the need for future work on multilingual reasoning that preserves reliability and trust, and provides open-source data and baselines to spur progress.

Abstract

Recent Large Reasoning Models (LRMs) with thinking traces have shown strong performance on English reasoning tasks. However, their ability to think in other languages is less studied. This capability is as important as answer accuracy for real world applications because users may find the reasoning trace useful for oversight only when it is expressed in their own language. We comprehensively evaluate two leading families of LRMs on our XReasoning benchmark and find that even the most advanced models often revert to English or produce fragmented reasoning in other languages, revealing a substantial gap in multilingual reasoning. Prompt based interventions that force models to reason in the users language improve readability and oversight but reduce answer accuracy, exposing an important trade off. We further show that targeted post training on just 100 examples mitigates this mismatch, though some accuracy loss remains. Our results highlight the limited multilingual reasoning capabilities of current LRMs and outline directions for future work. Code and data are available at https://github.com/Betswish/mCoT-XReasoning.

When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy

TL;DR

This work examines multilingual reasoning in large reasoning models and reveals a robust trade-off between forcing models to think in a user's language and maintaining answer accuracy. It introduces the XReasoning benchmark to evaluate thinking-language alignment across 11 languages on math and science tasks, and shows that prompt hacking can dramatically increase language matching at a cost to accuracy, with the effect diminishing as model size grows. Targeted post-training with a small number of in-language examples improves alignment but further degrades performance, while model merging offers partial gains in matching and limited accuracy improvements. The study highlights the need for future work on multilingual reasoning that preserves reliability and trust, and provides open-source data and baselines to spur progress.

Abstract

Recent Large Reasoning Models (LRMs) with thinking traces have shown strong performance on English reasoning tasks. However, their ability to think in other languages is less studied. This capability is as important as answer accuracy for real world applications because users may find the reasoning trace useful for oversight only when it is expressed in their own language. We comprehensively evaluate two leading families of LRMs on our XReasoning benchmark and find that even the most advanced models often revert to English or produce fragmented reasoning in other languages, revealing a substantial gap in multilingual reasoning. Prompt based interventions that force models to reason in the users language improve readability and oversight but reduce answer accuracy, exposing an important trade off. We further show that targeted post training on just 100 examples mitigates this mismatch, though some accuracy loss remains. Our results highlight the limited multilingual reasoning capabilities of current LRMs and outline directions for future work. Code and data are available at https://github.com/Betswish/mCoT-XReasoning.

Paper Structure

This paper contains 32 sections, 3 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multilingual Reasoning Benchmarks
  4. Improving Answer Accuracy with Cross-lingual Reasoning
  5. Experimental Setup
  6. Benchmarks
  7. Models
  8. Evaluation
  9. Experiments and Results
  10. Language Matching vs. Answer Accuracy
  11. Prompt Hacking
  12. Overall Results
  13. Language-Specific Results
  14. Actual Thinking Languages
  15. Post-Training with Few Instances
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Illustration of the trade-off between answer accuracy and thinking language matching in multilingual reasoning systems. English translations in brackets.
  • Figure 2: Heatmaps of language matching rates (top) and accuracy (bottom) for Distilled-R1-32B on AIME, before (left) and after (right) prompt hacking.
  • Figure 3: Language matching rate and answer accuracy of Distilled-R1-7B with no post-training, post-training on 100 instances, and post-training on 250 instances for 3 languages: Japanese, Thai, and Telugu.