Reasoning and Tool-use Compete in Agentic RL:From Quantifying Interference to Disentangled Tuning

TL;DR

This paper introduces a Linear Effect Attribution System (LEAS), which provides quantitative evidence of interference between reasoning and tool-use behaviors and proposes Disentangled Action Reasoning Tuning, a simple and efficient framework that explicitly decouples parameter updates for reasoning and tool-use via separate low-rank adaptation modules.

Abstract

Agentic Reinforcement Learning (ARL) focuses on training large language models (LLMs) to interleave reasoning with external tool execution to solve complex tasks. Most existing ARL methods train a single shared model parameters to support both reasoning and tool use behaviors, implicitly assuming that joint training leads to improved overall agent performance. Despite its widespread adoption, this assumption has rarely been examined empirically. In this paper, we systematically investigate this assumption by introducing a Linear Effect Attribution System(LEAS), which provides quantitative evidence of interference between reasoning and tool-use behaviors. Through an in-depth analysis, we show that these two capabilities often induce misaligned gradient directions, leading to training interference that undermines the effectiveness of joint optimization and challenges the prevailing ARL paradigm. To address this issue, we propose Disentangled Action Reasoning Tuning(DART), a simple and efficient framework that explicitly decouples parameter updates for reasoning and tool-use via separate low-rank adaptation modules. Experimental results show that DART consistently outperforms baseline methods with averaged 6.35 percent improvements and achieves performance comparable to multi-agent systems that explicitly separate tool-use and reasoning using a single model.

Figures (10)

• Figure 1: Overview of the Linear Effect Attribution System (LEAS). (A).Inference-derived Model: By routing specific token types to different models at inference time, we synthesize capability combinations without parameter-level interaction. (B).Linear Effect Attribution System: The six model variants (train-derived and inference-derived) populate the design matrix $\mathbf{X}$. Solving the linear system yields question-specific coefficients $\boldsymbol{\lambda}^q$, where $\lambda_{23}<0$ signals capability interference. (C).Token-Level Gradient Masking: During training, token-level masks selectively route gradients to reasoning or tool-use parameters, isolating capability-specific updates. (D).Training-derived Models: This produces specialized model variants derived from a shared backbone, enabling controlled comparisons across different capability combinations.
• Figure 2: Interaction between reasoning and tool-use under ARL. Histograms show the distribution of the question-level interaction coefficient $\lambda_{23}^q$ on NQ and HotpotQA using Qwen2.5-Instruct models (3B and 7B), where negative values (blue) indicate interference and positive values (red) indicate synergy. The overlaid curve shows ARL accuracy averaged over questions within each $\lambda_{23}^q$ bin.
• Figure 3: Gradient misalignment leads to optimization inefficiency.(A). Gradient angle distributions on NQ under Qwen2.5-3B, where same-capability gradients are aligned, while reasoning and tool-use gradients are nearly orthogonal. (B). Averaged orthogonal gradients yield a compromise update direction, leading to optimization inefficiency.
• Figure 4: Illustration of DART. A frozen backbone augmented with two disjoint LoRA adapters for reasoning and tool-use, both attached to all linear layers, where a token-level router directs gradients into separate parameter subspaces to avoid interference.
• Figure 5: Reasoning under Fixed Retrieval. DART achieves higher EM than Search-R1 on NQ and HotpotQA when both use identical retrieval contexts, demonstrating improved reasoning capability independent of retrieval quality.

Theorems & Definitions (2)

• Definition 1
• Definition 2