PRInTS: Reward Modeling for Long-Horizon Information Seeking

TL;DR

PRInTS tackles long-horizon information seeking by introducing a unified generative reward model that jointly learns dense step-level scoring and recursive trajectory summarization. It frames step evaluation as information gain estimation, training with reinforcement learning on Monte Carlo rollouts to produce multi-faceted scores, and uses summarization to bound context growth without losing essential evidence. The approach demonstrates substantial test-time gains across open-source and frontier LLMs on FRAMES, GAIA, and WebWalkerQA, and shows strong generalization to frontier models. This yields a practical, data-efficient path to improve information-seeking agents without requiring fine-tuning of underlying models, enabling scalable, robust decision-making in complex, multi-step tasks.

Abstract

Information-seeking is a core capability for AI agents, requiring them to gather and reason over tool-generated information across long trajectories. However, such multi-step information-seeking tasks remain challenging for agents backed by language models. While process reward models (PRMs) can guide agents by ranking candidate steps at test-time, existing PRMs, designed for short reasoning with binary judgment, cannot capture richer dimensions of information-seeking steps, such as tool interactions and reasoning over tool outputs, nor handle the rapidly growing context in long-horizon tasks. To address these limitations, we introduce PRInTS, a generative PRM trained with dual capabilities: (1) dense scoring based on the PRM's reasoning across multiple step quality dimensions (e.g., interpretation of tool outputs, tool call informativeness) and (2) trajectory summarization that compresses the growing context while preserving essential information for step evaluation. Extensive evaluations across FRAMES, GAIA (levels 1-3), and WebWalkerQA (easy-hard) benchmarks on multiple models, along with ablations, reveal that best-of-n sampling with PRInTS enhances information-seeking abilities of open-source models as well as specialized agents, matching or surpassing the performance of frontier models with a much smaller backbone agent and outperforming other strong reward modeling baselines.

Figures (10)

• Figure 1: Comparison between existing PRMs and PRInTS. Top: Existing PRMs are limited for long-horizon information-seeking as they evaluate a short reasoning unit (e.g., one-to-two-sentence inferences) with coarse feedback, which cannot capture multi-faceted quality factors from tool interactions. They also struggle with rapidly accumulating reasoning context (left). Bottom: In contrast, PRInTS evaluates a complete trajectory step (reasoning + tool interactions), considers multiple trajectory step quality dimensions to produce dense scores for finer-grained guidance at each step, and maintains compact trajectory summaries that keep key information for the evaluation.
• Figure 2: Overview of PRInTS. Left: PRInTS functions as a scorer, evaluating agent's multiple candidate next trajectory steps based on the summarized context and current tool response. It generates an analysis and a dense score for each candidate, selecting the top-scoring one to guide the agent's information-seeking. Right: PRInTS acts as a summarizer, recursively updating a compact information-seeking trajectory summary to keep input length bounded and preserve key information for its subsequent score evaluation.
• Figure 3: PRInTS: data annotation and training pipeline.Top: For each trajectory step, we estimate the information gain score via Monte Carlo rollouts as the change in mean answer accuracy before and after the step. Then we construct winning-losing step pairs based on these scores (left). Preference pair examples are shown in \ref{['fig:win_lose_examples']}. Then we train PRInTS as a scorer via GRPO on these pairs (right). The final reward combines a score reward for accurate prediction, a comparison reward for pairwise preference learning, and an adaptive weight to mitigate noisy annotations. Bottom: Each step is annotated with a compact, recursively updated trajectory summary capturing essential findings and plans up to the step (left). The same PRM is jointly trained as a summarizer via SFT on this summary data (right).
• Figure 4: Scaling test-time compute. Best-of-$n$ test-time scaling results on GAIA Level 2 using Qwen3-32B. PRInTS benefits from additional test-time compute by identifying higher-quality steps from $n$ candidates.
• Figure 5: Distribution of annotated information gain scores.