Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages

Anaelia Ovalle, Candace Ross, Sebastian Ruder, Adina Williams, Karen Ullrich, Mark Ibrahim, Levent Sagun

TL;DR

The paper tackles whether multilingual language models transfer reasoning abilities across languages by introducing a human-validated framework that decouples reasoning traces from final answers and evaluates cross-language trace support for conclusions. It applies this framework to GlobalMMLU across English, Spanish, Hindi, Arabic, Korean, and Ukrainian, using six frontier models and backtranslation-based quality control. The study reveals substantial reasoning–answer misalignment, especially for non-Latin scripts, and develops an automated evaluator validated against human judgments to enable scalable analysis. The findings highlight the need for reasoning-aware multilingual evaluation and point to predominant evidential errors as a key bottleneck, with implications for grounding, translation quality, and model calibration.

Abstract

Large language models demonstrate strong reasoning capabilities through chain-of-thought prompting, but whether this reasoning quality transfers across languages remains underexplored. We introduce a human-validated framework to evaluate whether model-generated reasoning traces logically support their conclusions across languages. Analyzing 65k reasoning traces from GlobalMMLU questions across 6 languages and 6 frontier models, we uncover a critical blind spot: while models achieve high task accuracy, their reasoning can fail to support their conclusions. Reasoning traces in non-Latin scripts show at least twice as much misalignment between their reasoning and conclusions than those in Latin scripts. We develop an error taxonomy through human annotation to characterize these failures, finding they stem primarily from evidential errors (unsupported claims, ambiguous facts) followed by illogical reasoning steps. Our findings demonstrate that current multilingual evaluation practices provide an incomplete picture of model reasoning capabilities and highlight the need for reasoning-aware evaluation frameworks.

Beg to Differ: Understanding Reasoning-Answer Misalignment Across Languages

TL;DR

The paper tackles whether multilingual language models transfer reasoning abilities across languages by introducing a human-validated framework that decouples reasoning traces from final answers and evaluates cross-language trace support for conclusions. It applies this framework to GlobalMMLU across English, Spanish, Hindi, Arabic, Korean, and Ukrainian, using six frontier models and backtranslation-based quality control. The study reveals substantial reasoning–answer misalignment, especially for non-Latin scripts, and develops an automated evaluator validated against human judgments to enable scalable analysis. The findings highlight the need for reasoning-aware multilingual evaluation and point to predominant evidential errors as a key bottleneck, with implications for grounding, translation quality, and model calibration.

Abstract

Large language models demonstrate strong reasoning capabilities through chain-of-thought prompting, but whether this reasoning quality transfers across languages remains underexplored. We introduce a human-validated framework to evaluate whether model-generated reasoning traces logically support their conclusions across languages. Analyzing 65k reasoning traces from GlobalMMLU questions across 6 languages and 6 frontier models, we uncover a critical blind spot: while models achieve high task accuracy, their reasoning can fail to support their conclusions. Reasoning traces in non-Latin scripts show at least twice as much misalignment between their reasoning and conclusions than those in Latin scripts. We develop an error taxonomy through human annotation to characterize these failures, finding they stem primarily from evidential errors (unsupported claims, ambiguous facts) followed by illogical reasoning steps. Our findings demonstrate that current multilingual evaluation practices provide an incomplete picture of model reasoning capabilities and highlight the need for reasoning-aware evaluation frameworks.
Paper Structure (27 sections, 2 figures, 10 tables)

This paper contains 27 sections, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multilingual reasoning evaluation
  4. Consistency and faithfulness in LLM reasoning
  5. Methodology
  6. Evaluation Framework
  7. Experimental Setup
  8. Data and Languages
  9. Models and Prompting
  10. Translation and Backtranslation Quality Control
  11. Evaluation Protocols
  12. Human Evaluation
  13. Automated Evaluation
  14. Evaluation Validation
  15. Human Evaluation Results
  16. ...and 12 more sections

Figures (2)

  • Figure 1: Example of model factuality failure in closed form setting, where the model's final answer is correct both in English and non-English example. however, the model gets the question right for the wrong reasons in non-English example.
  • Figure 2: Taxonomy distribution across models. Top is higher-resourced languages, bottom is lower-resourced languages.