ORBIT: On-policy Exploration-Exploitation for Controllable Multi-Budget Reasoning

TL;DR

ORBIT is proposed, a controllable multi-budget reasoning framework with well-separated reasoning modes triggered by input that achieves controllable reasoning behavior over multiple modes, competitive reasoning density within each mode, and integration of these frontier policies into a single unified student model while preserving clear mode separation and high per-mode performance.

Abstract

Recent Large Reasoning Models (LRMs) achieve strong performance by leveraging long-form Chain-of-Thought (CoT) reasoning, but uniformly applying overlong reasoning at inference time incurs substantial and often unnecessary computational cost. To address this, prior work explores various strategies to infer an appropriate reasoning budget from the input. However, such approaches are unreliable in the worst case, as estimating the minimal required reasoning effort is fundamentally difficult, and they implicitly fix the trade-off between reasoning cost and accuracy during training, limiting flexibility under varying deployment scenarios. Motivated by these limitations, we propose ORBIT, a controllable multi-budget reasoning framework with well-separated reasoning modes triggered by input. ORBIT employs multi-stage reinforcement learning to discover Pareto-optimal reasoning behaviors at each effort, followed by on-policy distillation to fuse these behaviors into a single unified model. Experiments show that ORBIT achieves (1) controllable reasoning behavior over multiple modes, (2) competitive reasoning density within each mode, and (3) integration of these frontier policies into a single unified student model while preserving clear mode separation and high per-mode performance.

Figures (7)

• Figure 1: ORBIT demonstrates controllable reasoning behavior comparable to advanced systems, covering multiple reasoning modes: low/mid/high for o3-mini and gpt-oss-120B, and an extended set of low/mid/high/extra-high for ORBIT.
• Figure 2: The ORBIT framework overview: (1) In the Exploration stage, multi-stage RL discovers specialized reasoning frontiers under varying context from the expansion-compression loop. (2) In the Exploitation stage, these behaviors are unified into a single student model via model merging and multi-teacher on-policy distillation.
• Figure 3: Step-wise performance of ORBIT and joint RL on AIME24, with steps aligned by total training tokens for fair comparison. ORBIT starts from step 278, where OPD training begins. Sequential compression stages are omitted for clarity.
• Figure 4: Comparison of OPD and offline distillation as the fusing strategy on AIME24, with steps aligned by total training samples for fair comparison.On-policy fusion achieves slightly higher performance while both strategies show similar trend.
• Figure 5: Cognition analysis of ORBIT across reasoning modes, vertical axis denotes average count per sample.