AdaTIR: Adaptive Tool-Integrated Reasoning via Difficulty-Aware Policy Optimization

TL;DR

AdaTIR tackles cognitive offloading in Tool-Integrated Reasoning by introducing a difficulty-aware policy that adaptively budgets tool calls based on task complexity. It couples a sine-based Difficulty-Aware Reward with Clipped Advantage Shaping to maintain correctness as the primary objective while reducing unnecessary tool usage. Empirical results show AdaTIR achieves strong accuracy with substantial reductions in tool calls (up to 97.6% on simple tasks and 28.2% on complex tasks) and even internalizes reasoning when tool access is disabled. The approach yields robust improvements on mathematical benchmarks and demonstrates superior reasoning capability and efficiency compared with prior RL-based baselines and OTC/ToRL variants.

Abstract

Tool-Integrated Reasoning (TIR) has significantly enhanced the capabilities of Large Language Models (LLMs), yet current agents tend to exhibit cognitive offloading, redundantly invoking external tools even for simple tasks. In this paper, we suggest that true agentic intelligence requires not just tool invocation, but the adaptive wisdom to discern when to use them. We propose AdaTIR, a framework that shifts the paradigm from static tool invocation to difficulty-aware reasoning internalization. By introducing a difficulty-aware efficiency reward, AdaTIR dynamically adjusts tool budgets based on task complexity--internalizing reasoning for simple tasks while selectively invoking tools for complex tasks. Furthermore, we identify a sign reversal problem where tool penalties outweigh correctness rewards, mistakenly penalizing correct rollouts with negative advantages. To resolve this, we propose Clipped Advantage Shaping (CAS), which ensures that correctness remains the primary objective while using efficiency as a secondary constraint. Empirical results demonstrate that AdaTIR reduces tool calls by up to 97.6% on simple tasks and 28.2% on complex challenges while maintaining or enhancing accuracy. Notably, AdaTIR successfully internalizes reasoning, outperforming baselines by 4.8% on AIME 2024 even when tool access is strictly disabled.

Figures (7)

• Figure 1: Accuracy versus Tool Calls for Qwen2.5 under different training strategies. GRPO and AdaTIR denote the baseline and our proposed method, respectively. AdaTIR consistently achieves a superior efficiency-accuracy trade-off across both 3B and 7B backbones by internalizing reasoning capabilities.
• Figure 2: Overview of the proposed framework. Efficiency signals are conditionally injected only for correct rollouts on easy tasks (where difficulty estimation $\phi$ falls below threshold $\phi_{low}$). These auxiliary signals are explicitly clipped by $\delta |A_{acc}|$ ($0 < \delta < 1$), to ensure efficiency optimization remains secondary to correctness, thereby maintaining sign consistency.
• Figure 3: Comparison of reasoning processes between a standard tool-integrated baseline and our approach. (Left) The baseline model relies on external Python tools but fails to recover after the symbolic solver returns an error. It then attempts an irrelevant search and ends with an incorrect answer. (Right) Our approach correctly solves the problem by performing step-by-step algebraic deduction internally, reaching the correct result without depending on external code execution.
• Figure 4: Reasoning accuracy and tool invocation under varying tool budgets $B$. The horizontal axis denotes the maximum allowed tool calls, ranging from B0 ($B=0$) to B4 ($B=4$). The top row presents the reasoning accuracy (reported as Avg@16), while the bottom row shows the Average Tool Calls (ATC). Our method consistently outperforms the baseline across all budget levels, achieving higher accuracy with significantly fewer tool invocations. Notably, even at B0 where no tool invocation is permitted, our method exhibits superior performance; this indicates that our approach facilitates the internalization of reasoning capabilities and mitigates cognitive offloading, enabling the model to maintain robust performance even without external tool calls
• Figure 5: Training stability analysis of AdaTIR compared with standard reward shaping. From left to right, the panels display the Gradient Norm, Training Reward, and Response Length over training steps. Solid lines represent the smoothed curves using exponential moving average (EMA), while the semi-transparent shaded areas indicate the raw fluctuations.