ASTER: Agentic Scaling with Tool-integrated Extended Reasoning

TL;DR

ASTER tackles interaction collapse in tool-integrated reinforcement learning by introducing an interaction-dense cold-start prior of just $4K$ tool-rich trajectories to preserve exploration across extended GRPO training. It achieves state-of-the-art results on competitive math benchmarks, including a score of $90.0\%$ on AIME $2025$ under large inference budgets, with a 4B model rivaling much larger systems. The approach combines a two-stage RL curriculum with a curated tool-augmented dataset and an emphasis on long-horizon planning, enabling sustained multi-turn tool use and robust generalization. These results highlight the importance of behavioral priors and budget alignment for scalable agentic reasoning and discuss safety considerations for tool integration.

Abstract

Reinforcement learning (RL) has emerged as a dominant paradigm for eliciting long-horizon reasoning in Large Language Models (LLMs). However, scaling Tool-Integrated Reasoning (TIR) via RL remains challenging due to interaction collapse: a pathological state where models fail to sustain multi-turn tool usage, instead degenerating into heavy internal reasoning with only trivial, post-hoc code verification. We systematically study three questions: (i) how cold-start SFT induces an agentic, tool-using behavioral prior, (ii) how the interaction density of cold-start trajectories shapes exploration and downstream RL outcomes, and (iii) how the RL interaction budget affects learning dynamics and generalization under varying inference-time budgets. We then introduce ASTER (Agentic Scaling with Tool-integrated Extended Reasoning), a framework that circumvents this collapse through a targeted cold-start strategy prioritizing interaction-dense trajectories. We find that a small expert cold-start set of just 4K interaction-dense trajectories yields the strongest downstream performance, establishing a robust prior that enables superior exploration during extended RL training. Extensive evaluations demonstrate that ASTER-4B achieves state-of-the-art results on competitive mathematical benchmarks, reaching 90.0% on AIME 2025, surpassing leading frontier open-source models, including DeepSeek-V3.2-Exp.

Figures (15)

• Figure 1: ASTER demonstrates remarkable efficiency, surpassing much larger and stronger models on the challenging AIME 2025 benchmark. It achieves a score of 90.0, outperforming DeepSeek-V3.2-exp (89.3/671B).
• Figure 2: Distributional properties of cold-start datasets. (a) Tool-call count distribution reveals that ReTool and DemyAgent datasets are heavily biased toward sparse interactions (1--2 tool calls), while our dataset contains more long-horizon trajectories. (b) Different cold-start strategies induce distinct behavioral priors, using agentic judger to evaluate their agentic capabilities.
• Figure 3: RQ1 analysis: Cold-start strategies shape downstream RL dynamics. (a) and (b) illustrate the impact of the synthetic teacher patterns, showing performance gaps and distinct tool-use strategies.
• Figure 4: Tool intensive SFT achieves higher performance on AIME25.
• Figure 5: Test-time performance scaling as a function of inference-time tool budget for models trained under different interaction constraints. Models trained with higher interaction budgets (50 calls) dominate under large inference budgets, while constrained-training models perform better when inference is severely limited.