Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning

TL;DR

The paper addresses the challenge of reliably evaluating LLM outputs by moving beyond text-only reasoning to tool-grounded judgments. It introduces TIR-Judge, a tool-integrated reinforcement learning framework that jointly learns reasoning and code execution to produce verifiable evaluations across pointwise, pairwise, and listwise settings. Key contributions include a diverse data mixture of verifiable/non-verifiable tasks, a structured reward design with correctness, formatting, and tool-use components, and stabilization techniques for tool-augmented RL. Empirical results on seven benchmarks show substantial gains over strong baselines, with 8B-parameter TIR-Judge achieving near state-of-the-art listwise performance and a zero-shot variant capable of self-improvement without distillation, highlighting practical impact for scalable, self-improving LLM judges in evaluation pipelines.

Abstract

Large Language Models (LLMs) are widely used as judges to evaluate response quality, providing a scalable alternative to human evaluation. However, most LLM judges operate solely on intrinsic text-based reasoning, limiting their ability to verify complex constraints or perform accurate computation. Motivated by the success of tool-integrated reasoning (TIR) in numerous tasks, we propose TIR-Judge, an end-to-end RL framework for training LLM judges that integrates a code executor for precise evaluation. TIR-Judge is built on three principles: (i) diverse training across verifiable and non-verifiable domains, (ii) flexible judgment formats (pointwise, pairwise, listwise), and (iii) iterative RL that bootstraps directly from the initial model without distillation. On seven public benchmarks, TIR-Judge surpasses strong reasoning-based judges by up to 6.4% (pointwise) and 7.7% (pairwise), and achieves listwise performance comparable to Claude-Opus-4 despite having only 8B parameters. Remarkably, TIR-Judge-Zero - trained entirely without distilled judge trajectories, matches the performance of distilled variants, demonstrating that tool-augmented judges can self-evolve through iterative reinforcement learning.

Figures (12)

• Figure 1: An example of LLM judge augmented with code execution, enabling precise judgments.
• Figure 2: Overall framework of TIR-Judge variants. TIR-Judge natively supports tool use during judgment and is designed to handle diverse input formats.
• Figure 3: The effect of different data mixture used in RL training of TIR-Judge-Zero.
• Figure 4: Experimental results comparing tool-augmented judges against text-only judges under the same training data and settings, as well as the best-of-$N$ inference performance.
• Figure 5: Accuracy of TIR-Judge across different training stages. Base denotes the backbone model without additional training. TIR-Judge-Zero-RS is a variant used in zelikman2022star that uses rejection sampling to construct high-quality trajectories for SFT (without RL). TIR-Judge-Zero-RL-0 refer to the judge with direct RL training, and TIR-Judge-Zero-RL-0 refer to the performance of TIR-Judge after 1, and 2 iterations of RS-SFT-RL cycles, respectively.