Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ExpLang: Improved Exploration and Exploitation in LLM Reasoning with On-Policy Thinking Language Selection

Changjiang Gao, Zixian Huang, Kaichen Yang, Jiajun Chen, Jixing Li, Shujian Huang

TL;DR

Analysis shows that, by enabling on-policy thinking language selection as an action during RL, ExpLang effectively extends the RL exploration space with diversified language preference and improves the RL exploitation outcome with leveraged non-English advantage.

Abstract

Current large reasoning models (LRMs) have shown strong ability on challenging tasks after reinforcement learning (RL) based post-training. However, previous work mainly focuses on English reasoning in expectation of the strongest performance, despite the demonstrated potential advantage of multilingual thinking, as well as the requirement for native thinking traces by global users. In this paper, we propose ExpLang, a novel LLM post-training pipeline that enables on-policy thinking language selection to improve exploration and exploitation during RL with the use of multiple languages. The results show that our method steadily outperforms English-only training with the same training budget, while showing high thinking language compliance for both seen and unseen languages. Analysis shows that, by enabling on-policy thinking language selection as an action during RL, ExpLang effectively extends the RL exploration space with diversified language preference and improves the RL exploitation outcome with leveraged non-English advantage. The method is orthogonal to most RL algorithms and opens up a new perspective on using multilinguality to improve LRMs.

ExpLang: Improved Exploration and Exploitation in LLM Reasoning with On-Policy Thinking Language Selection

TL;DR

Analysis shows that, by enabling on-policy thinking language selection as an action during RL, ExpLang effectively extends the RL exploration space with diversified language preference and improves the RL exploitation outcome with leveraged non-English advantage.

Abstract

Current large reasoning models (LRMs) have shown strong ability on challenging tasks after reinforcement learning (RL) based post-training. However, previous work mainly focuses on English reasoning in expectation of the strongest performance, despite the demonstrated potential advantage of multilingual thinking, as well as the requirement for native thinking traces by global users. In this paper, we propose ExpLang, a novel LLM post-training pipeline that enables on-policy thinking language selection to improve exploration and exploitation during RL with the use of multiple languages. The results show that our method steadily outperforms English-only training with the same training budget, while showing high thinking language compliance for both seen and unseen languages. Analysis shows that, by enabling on-policy thinking language selection as an action during RL, ExpLang effectively extends the RL exploration space with diversified language preference and improves the RL exploitation outcome with leveraged non-English advantage. The method is orthogonal to most RL algorithms and opens up a new perspective on using multilinguality to improve LRMs.
Paper Structure (33 sections, 1 equation, 6 figures, 8 tables)

This paper contains 33 sections, 1 equation, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. From Chain-of-Thoughts to RL-Enhanced Reasoning
  4. Multilingual Reasoning of LLMs
  5. Building Blocks
  6. High-Quality Multilingual Thinking Data
  7. Cold-Start of Thinking Language Selection
  8. RLVR with Thinking Language Selection
  9. Experiment Settings
  10. Models, Data and Training Strategies
  11. Baselines
  12. Main Results
  13. Potential Benefit of Multilingual Thinking
  14. Performance with Automatic Selection
  15. Performance with Forced Selection
  16. ...and 18 more sections

Figures (6)

  • Figure 1: Overview of our work. a. Multilingual thinking shows larger diversity and higher Pass@$k$, suggesting better exploration space; b. The ExpLang pipeline with on-policy thinking language selection to leverage the advantage.
  • Figure 2: Trends of thinking language selection rates of our model at the end of different training stages, across the three test sets.
  • Figure 3: Policy entropy of ExpLang and the controlled baseline during the RLVR process.
  • Figure 4: Win-Tie-Lose rate of non-English vs. English thinking, in terms of the average accuracy of the models on 2000 training samples, run for 5 times per sample.
  • Figure 5: Comparison of Cluster Counts After Spectral Clustering of English and Multilingual Thinking.
  • ...and 1 more figures