Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Think Natively: Unlocking Multilingual Reasoning with Consistency-Enhanced Reinforcement Learning

Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Kaiyu Huang, Yufeng Chen, Jinan Xu, Jie Zhou

TL;DR

The paper tackles multilingual reasoning in large reasoning models by addressing input-output language inconsistency and weaker non-English reasoning. It introduces M-Thinker, a GRPO-trained model that uses a Language Consistency reward to enforce language fidelity and a Cross-lingual Thinking Alignment reward to transfer English reasoning to other languages. Experiments on MMATH and PolyMath demonstrate near-ideal language consistency and strong multilingual accuracy, with notable out-of-domain generalization. The authors present a scalable training pipeline—comprising cold-start SFT, rejection sampling, and iterative RL—to show that language-consistent multilingual reasoning can approach English-level performance across multiple languages.

Abstract

Large Reasoning Models (LRMs) have achieved remarkable performance on complex reasoning tasks by adopting the "think-then-answer" paradigm, which enhances both accuracy and interpretability. However, current LRMs exhibit two critical limitations when processing non-English languages: (1) They often struggle to maintain input-output language consistency; (2) They generally perform poorly with wrong reasoning paths and lower answer accuracy compared to English. These limitations significantly degrade the user experience for non-English speakers and hinder the global deployment of LRMs. To address these limitations, we propose M-Thinker, which is trained by the GRPO algorithm that involves a Language Consistency (LC) reward and a novel Cross-lingual Thinking Alignment (CTA) reward. Specifically, the LC reward defines a strict constraint on the language consistency between the input, thought, and answer. Besides, the CTA reward compares the model's non-English reasoning paths with its English reasoning path to transfer its own reasoning capability from English to non-English languages. Through an iterative RL procedure, our M-Thinker-1.5B/7B models not only achieve nearly 100% language consistency and superior performance on two multilingual benchmarks (MMATH and PolyMath), but also exhibit excellent generalization on out-of-domain languages.

Think Natively: Unlocking Multilingual Reasoning with Consistency-Enhanced Reinforcement Learning

TL;DR

The paper tackles multilingual reasoning in large reasoning models by addressing input-output language inconsistency and weaker non-English reasoning. It introduces M-Thinker, a GRPO-trained model that uses a Language Consistency reward to enforce language fidelity and a Cross-lingual Thinking Alignment reward to transfer English reasoning to other languages. Experiments on MMATH and PolyMath demonstrate near-ideal language consistency and strong multilingual accuracy, with notable out-of-domain generalization. The authors present a scalable training pipeline—comprising cold-start SFT, rejection sampling, and iterative RL—to show that language-consistent multilingual reasoning can approach English-level performance across multiple languages.

Abstract

Large Reasoning Models (LRMs) have achieved remarkable performance on complex reasoning tasks by adopting the "think-then-answer" paradigm, which enhances both accuracy and interpretability. However, current LRMs exhibit two critical limitations when processing non-English languages: (1) They often struggle to maintain input-output language consistency; (2) They generally perform poorly with wrong reasoning paths and lower answer accuracy compared to English. These limitations significantly degrade the user experience for non-English speakers and hinder the global deployment of LRMs. To address these limitations, we propose M-Thinker, which is trained by the GRPO algorithm that involves a Language Consistency (LC) reward and a novel Cross-lingual Thinking Alignment (CTA) reward. Specifically, the LC reward defines a strict constraint on the language consistency between the input, thought, and answer. Besides, the CTA reward compares the model's non-English reasoning paths with its English reasoning path to transfer its own reasoning capability from English to non-English languages. Through an iterative RL procedure, our M-Thinker-1.5B/7B models not only achieve nearly 100% language consistency and superior performance on two multilingual benchmarks (MMATH and PolyMath), but also exhibit excellent generalization on out-of-domain languages.

Paper Structure

This paper contains 44 sections, 9 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Background: GRPO
  5. Reward Modeling
  6. Language Consistency Reward.
  7. Cross-lingual Thinking Alignment Reward.
  8. Format Reward.
  9. Accuracy Reward.
  10. Overall Reward.
  11. Training Procedure
  12. Experiments
  13. Experimental Setups
  14. Backbones and Languages.
  15. Benchmarks and Metrics.
  16. ...and 29 more sections

Figures (4)

  • Figure 1: Existing LRMs struggle to maintain input-output language consistency and probably offer us the wrong answer when processing non-English inputs, while our M-Thinker can respond in the input language with the correct answer.
  • Figure 2: The language control instructions wang2025polymath of the Prompt-Control baseline.
  • Figure 3: The multilingual discourse marks for each language luo2025mmathmultilingualbenchmarkmathematical of the DIT baseline.
  • Figure 4: The restatement instructions luo2025mmathmultilingualbenchmarkmathematical of the QRT baseline.