Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents

TL;DR

PROBE offers an end-to-end benchmark for proactive AI agents by formalizing proactivity as a three-stage pipeline (search for issues, bottleneck identification, and action execution) and constructing a large synthetic datastore with realistic bottlenecks. The data-generation pipeline builds comprehensive world models from professional personas, injects bottlenecks, and distributes evidence across multiple documents to require cross-document reasoning. Evaluations across frontier and open-source models, as well as agent frameworks, show a persistent capability ceiling with end-to-end success around 0.4, driven by bottleneck identification and parameter specification challenges. The work also provides extensive ablations and error analyses, revealing context size sensitivity and the value of multi-model data generation to mitigate model-specific artifacts. Overall, PROBE highlights significant gaps in truly proactive AI systems and outlines concrete directions for improving world modeling, long-horizon evidence use, and end-to-end evaluation in real-world-like settings.

Abstract

LLM-based agents are increasingly moving towards proactivity: rather than awaiting instruction, they exercise agency to anticipate user needs and solve them autonomously. However, evaluating proactivity is challenging; current benchmarks are constrained to localized context, limiting their ability to test reasoning across sources and longer time horizons. To address this gap, we present PROBE (Proactive Resolution Of BottlEnecks). PROBE decomposes proactivity as a pipeline of three core capabilities: (1) searching for unspecified issues, (2) identifying specific bottlenecks, and (3) executing appropriate resolutions. We apply PROBE to evaluate leading LLMs and popular agentic frameworks, showing that even state-of-the-art models struggle to solve this benchmark. Computing our consistent measurements across frontier LLMs and agents, we find that the best end-to-end performance of 40% is achieved by both GPT-5 and Claude Opus-4.1. Additionally, we demonstrate the relative capabilities of each model and analyze mutual failure modes. Our results highlight the current limitations of autonomous action in agentic systems, and expose promising future research directions.

Figures (4)

• Figure 1: An end to end depiction of the PROBE task setup. The model (or agent) needs to use the world model to (i) search over a user datastore, (ii) identify the bottleneck and finally (iii) select the action to be executed. The model is evaluated across all tasks in this pipeline.
• Figure 2: End-to-end proactive benchmark generation pipeline. A synthetic world model ($W$) is constructed in reference to a LinkedIn user profile, followed by generating a bottleneck ($b$) to resolve. True Positives ($T\subseteq D$) and distractors (other documents unrelated to the bottleneck, $K = D\setminus T$) are then constructed to frame the bottleneck prediction task. Finally, a task to execute is selected with parameters ( $\mathcal{P}=\{P_a\}_{a\in A}$ for action $a\in A$ ) to resolve the bottleneck. For any bottleneck, only one of the candidate tasks is a plausible resolution.
• Figure 3: Performance degradation with increasing context window size. All models show declining performance as distractor count increases from 50 (baseline) to 100 (+50), with Claude models exhibiting steeper decay than GPT-5. This demonstrates the fundamental challenge of maintaining search and reasoning accuracy in long context, high-distractor environments.
• Figure 4: GPT-5 performance on data generated by different models. GPT-5 exhibits much higher search performance on samples generated by its own model family (GPT-5-mini: 0.951 F1, GPT-4.1: 0.882 F1) compared to cross-family generation (Claude Sonnet 4: 0.564 F1), validating our multi-model data generation approach.