Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning

TL;DR

Nemotron-CrossThink tackles the challenge of extending RL-based self-learning beyond math by leveraging diverse, multi-domain corpora. It introduces a scalable framework that integrates data curation from synthetic and open-source sources, structured templates to constrain output space, lightweight filtering, and data blending, all guided by group relative policy optimization and rule-based rewards. Empirical results show substantial gains on both math and non-math benchmarks, along with notable improvements in token efficiency for correct responses. The work demonstrates that carefully curated data diversity and formatting strategies are key to achieving robust, generalizable, and efficient reasoning in LLMs under reinforcement learning.

Abstract

Large Language Models (LLMs) have shown strong reasoning capabilities, particularly when enhanced through Reinforcement Learning (RL). While prior work has successfully applied RL to mathematical reasoning -- where rules and correctness are well-defined -- generalizing these methods to broader reasoning domains remains challenging due to limited data, the lack of verifiable reward structures, and diverse task requirements. In this work, we propose NEMOTRON-CROSSTHINK, a framework that systematically incorporates multi-domain corpora, including both synthetic and real-world question-answer pairs, into RL training to improve generalization across diverse reasoning tasks. NEMOTRON-CROSSTHINK addresses key challenges by (1) incorporating data from varied sources spanning STEM, humanities, social sciences, etc.; (2) applying structured templates (e.g., multiple-choice and open-ended) to control answer-space complexity; (3) filtering for verifiable answers; and (4) optimizing data blending strategies that utilizes data from multiple sources effectively. Our approach enables scalable and verifiable reward modeling beyond mathematics and demonstrates improved accuracies on both math (MATH-500: +30.1%, AMC23:+27.5%) and non-math reasoning benchmarks (MMLU-PRO: +12.8%, GPQA-DIAMOND: +11.3%, AGIEVAL: +15.1%, SUPERGPQA: +3.8%). Moreover, NEMOTRON-CROSSTHINK exhibits significantly improved response efficiency -- using 28% fewer tokens for correct answers -- highlighting more focused and effective reasoning. Through NEMOTRON-CROSSTHINK, we demonstrate that integrating multi-domain, multi-format data in RL leads to more accurate, efficient, and generalizable LLMs.

Figures (7)

• Figure 1: Employing self-learning with multi-domain data, Nemotron-CrossThink outperforms baseline models, including domain-specific training (Only Math) and Open-Reasoner-Zero (orz-7B), achieving consistent gains across all reasoning tasks.
• Figure 2: Nemotron-CrossThink. We (a) curate QA pairs from from synthetic (Common Crawl) and open-source datasets, categorized into general-purpose reasoning ($\mathcal{D}_{gpr}$) and mathematical reasoning ($\mathcal{D}_{mr}$); (b) apply structured templates to convert data into multiple-choice (mcq) and open-ended formats, promoting diverse reasoning trajectories; (c) filter out unverifiable or ill-formatted responses; (d) train an RL policy using Group Relative Policy Optimization (grpo). The final reward is used to update the policy, iteratively improving the model’s reasoning capabilities across diverse domains.
• Figure 3: Token efficiency comparison of models trained on $\mathcal{B}_{gpr\uparrow}$ (multi-domain blend) and two single domain blends ($\mathcal{B}_{only\_math}$ and orz).
• Figure 4: Average token lengths of correct and incorrect responses across general-purpose and math reasoning tasks for models trained on $\mathcal{B}_{gpr\uparrow}$, $\mathcal{B}_{only\_math}$, and orz.
• Figure 5: Sub-category Accuracy Comparison across mmlu-pro Domains. The $\mathcal{B}_{gpr\uparrow}$ blend consistently outperforms $\mathcal{B}_{only\_math}$ in a wide range of non-math reasoning categories such as business, law, psychology, and economics. Surprisingly, it also slightly surpasses the math-specialized blend in the mmlu-pro math category, highlighting the generalizability and versatility of multi-domain training.