A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages

TL;DR

The paper conducts the first large-scale, multilingual evaluation of Chain-of-Thought reasoning across performance, consistency, and faithfulness using diverse LRMs and two language-control methods. It introduces crosslingual thinking trace interchanging to assess semantic consistency of traces and uses perturbation-based methods (truncation and error injection) to probe trace faithfulness. Key findings show strong language preferences in reasoning, variable trace quality across languages, and persistent cross-language performance gaps, with trace control sometimes improving compliance at the cost of accuracy. The work demonstrates that thinking traces are not uniformly reliable across languages, highlights the influence of prompt language and model scale, and provides code and data to advance future multilingual CoT research.

Abstract

Large reasoning models (LRMs) increasingly rely on step-by-step Chain-of-Thought (CoT) reasoning to improve task performance, particularly in high-resource languages such as English. While recent work has examined final-answer accuracy in multilingual settings, the thinking traces themselves, i.e., the intermediate steps that lead to the final answer, remain underexplored. In this paper, we present the first comprehensive study of multilingual CoT reasoning, evaluating three key dimensions: performance, consistency, and faithfulness. We begin by measuring language compliance, answer accuracy, and answer consistency when LRMs are explicitly instructed or prompt-hacked to think in a target language, revealing strong language preferences and divergent performance across languages. Next, we assess crosslingual consistency of thinking traces by interchanging them between languages. We find that the quality and effectiveness of thinking traces vary substantially depending on the prompt language. Finally, we adapt perturbation-based techniques -- i.e., truncation and error injection -- to probe the faithfulness of thinking traces across languages, showing that models rely on traces to varying degrees. We release our code and data to support future research.

Figures (9)

• Figure 1: Final-answer consistency for R1-Qwen-32B and R1-Llama-70B under explicit instruction and prompt hacking. Similar language pairs, such as German and English, show higher consistency. Each cell shows the final-answer consistency between the language on the x-axis and the language on the y-axis.
• Figure 2: Final-answer accuracy of R1-Qwen-14B model under three thinking trace substitutions: BaseSub, HackSub, and TransSub. Each cell shows the accuracy when injecting thinking traces from a language on the y-axis into a language on the y-axis. Performance disparities indicate that thinking trace quality varies across languages.
• Figure 3: Substitution consistency of R1-Qwen-14B model under three thinking trace substitutions: BaseSub, HackSub, and TransSub. Each cell indicates the consistency between the original predictions in the language on the x-axis and the predictions after injecting thinking traces from the language on the y-axis. Higher consistency is observed when traces are substituted between similar languages.
• Figure 4: Mean accuracy drop (percentage) across languages for R1 distilled models under truncation of different parts of the thinking trace: first, middle, or last.
• Figure 5: Final-answer consistency heatmaps on the MGSM dataset across different models under explicit instruction and prompt hacking.