MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matching

TL;DR

MatchTIR tackles the challenge of coarse credit assignment in Tool-Integrated Reasoning by introducing turn-level supervision via bipartite matching and a dual-level advantage that combines local turn-specific feedback with global trajectory outcomes. It builds a similarity-based matching between predicted and ground-truth tool calls, yielding dense turn-level rewards through hard (Hungarian) or soft (optimal transport) alignments, and integrates an outcome-based reward to guide global success. The method uses a GRPO-based policy objective with a fused trajectory- and turn-level advantage to train the agent. Across in-domain and out-of-domain benchmarks, MatchTIR achieves state-of-the-art performance, with 4B models rivaling or surpassing many 8B baselines, particularly on long-horizon tasks.

Abstract

Tool-Integrated Reasoning (TIR) empowers large language models (LLMs) to tackle complex tasks by interleaving reasoning steps with external tool interactions. However, existing reinforcement learning methods typically rely on outcome- or trajectory-level rewards, assigning uniform advantages to all steps within a trajectory. This coarse-grained credit assignment fails to distinguish effective tool calls from redundant or erroneous ones, particularly in long-horizon multi-turn scenarios. To address this, we propose MatchTIR, a framework that introduces fine-grained supervision via bipartite matching-based turn-level reward assignment and dual-level advantage estimation. Specifically, we formulate credit assignment as a bipartite matching problem between predicted and ground-truth traces, utilizing two assignment strategies to derive dense turn-level rewards. Furthermore, to balance local step precision with global task success, we introduce a dual-level advantage estimation scheme that integrates turn-level and trajectory-level signals, assigning distinct advantage values to individual interaction turns. Extensive experiments on three benchmarks demonstrate the superiority of MatchTIR. Notably, our 4B model surpasses the majority of 8B competitors, particularly in long-horizon and multi-turn tasks. Our codes are available at https://github.com/quchangle1/MatchTIR.

Figures (5)

• Figure 1: Comparison of reward and advantage assignment strategies for multi-turn TIR. (a) and (b) show traditional methods where all reasoning steps share the same reward and advantage. (c) and (d) illustrate our MatchTIR framework using hard and soft assignment to derive turn-level rewards and distinct advantages.
• Figure 2: The illustration of our proposed MatchTIR. (a) illustrates the process of multi-turn TIR, where the policy LLM interacts with external tools over multiple turns and receives an outcome reward. (b) shows turn-level reward modeling by matching predicted and golden tool calls using hard or soft assignment strategies. (c) depicts the dual-level advantage estimation that integrates trajectory-level and turn-level signals for policy optimization.
• Figure 3: Performance comparison on subsets of varying task complexity, where queries are grouped based on the number of tools required to solve each task.
• Figure 4: Sensitivity analysis of performance to hyper-parameters. (a) shows the impact of penalty scales $\lambda$. (b) illustrates the effect of discounted factor $\gamma$.
• Figure 5: Prompt template used in our experiments.