CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning

TL;DR

CPMobius introduces a data-free, cooperative Coach–Player framework for improving mathematical reasoning in LLMs, decoupling progress from human-curated data. The Coach designs a curriculum tailored to the Player’s evolving capabilities, while the Player solves tasks and provides learning signals via verifiable outcomes and GRPO-based updates; rewards are shaped to maximize genuine skill growth. Across six benchmarks and multiple base models, CPMöbius achieves strong gains and notable out-of-distribution generalization, outperforming existing unsupervised RL methods and demonstrating robustness to different starting points. The approach offers a scalable alternative to supervised fine-tuning and adversarial self-play, with potential applications beyond mathematical reasoning and a clear pathway for future multi-domain collaboration studies.

Abstract

Large Language Models (LLMs) have demonstrated strong potential in complex reasoning, yet their progress remains fundamentally constrained by reliance on massive high-quality human-curated tasks and labels, either through supervised fine-tuning (SFT) or reinforcement learning (RL) on reasoning-specific data. This dependence renders supervision-heavy training paradigms increasingly unsustainable, with signs of diminishing scalability already evident in practice. To overcome this limitation, we introduce CPMöbius (CPMobius), a collaborative Coach-Player paradigm for data-free reinforcement learning of reasoning models. Unlike traditional adversarial self-play, CPMöbius, inspired by real world human sports collaboration and multi-agent collaboration, treats the Coach and Player as independent but cooperative roles. The Coach proposes instructions targeted at the Player's capability and receives rewards based on changes in the Player's performance, while the Player is rewarded for solving the increasingly instructive tasks generated by the Coach. This cooperative optimization loop is designed to directly enhance the Player's mathematical reasoning ability. Remarkably, CPMöbius achieves substantial improvement without relying on any external training data, outperforming existing unsupervised approaches. For example, on Qwen2.5-Math-7B-Instruct, our method improves accuracy by an overall average of +4.9 and an out-of-distribution average of +5.4, exceeding RENT by +1.5 on overall accuracy and R-zero by +4.2 on OOD accuracy.

Figures (5)

• Figure 1: CPMöbius starts with the coach proposing tasks of suitable difficulty. The player learns by solving these tasks, then reviews on a predefined environment. Finally, the coach adjusts the next training plan based on the player’s performance.
• Figure 3: Visualization of the Player's answer consistency on Coach proposed tasks during training. A lower value indicates higher difficulty of the instructions.
• Figure 4: Visualization of the training dynamics of CPMöbius using validation results on AMC dataset. The curves are smoothed with Time Weighted EMA, where CPMöbius shows consistent performance improvement for different base models.
• Figure 5: Visualization of the training dynamics on CPMöbius and different ablation experiments using validation results on AMC dataset.
• Figure 6: Visualization of the growing trend of output length of different models.