The PROPER Approach to Proactivity: Benchmarking and Advancing Knowledge Gap Navigation

TL;DR

The paper reframes proactive AI assistance as an epistemic calibration problem, introducing Proper, a modular framework that separates knowledge-gap discovery (DGA) from response generation (RGA) and uses a post-hoc reranker to selectively activate implicit dimensions. Across medical, coding, and shopping domains, ProPer improves task utility over strong baselines and CoT prompting, with larger gains in uncertain or risk-laden tasks. The work contributes a clean formalization of interaction dimensions, a dimension-level supervision dataset (ProPerBench), and a practical, calibrated architecture that demonstrates stable benefits in single-turn and multi-turn interactions. Its implications reach beyond surface-level correctness to trust, timing, and user intent alignment in proactive assistants, pointing to future extensions in adaptive calibration, concept-grounded representations, and multimodal signals.

Abstract

Most language-based assistants follow a reactive ask-and-respond paradigm, requiring users to explicitly state their needs. As a result, relevant but unexpressed needs often go unmet. Existing proactive agents attempt to address this gap either by eliciting further clarification, preserving this burden, or by extrapolating future needs from context, often leading to unnecessary or mistimed interventions. We introduce ProPer, Proactivity-driven Personalized agents, a novel two-agent architecture consisting of a Dimension Generating Agent (DGA) and a Response Generating Agent (RGA). DGA, a fine-tuned LLM agent, leverages explicit user data to generate multiple implicit dimensions (latent aspects relevant to the user's task but not considered by the user) or knowledge gaps. These dimensions are selectively filtered using a reranker based on quality, diversity, and task relevance. RGA then balances explicit and implicit dimensions to tailor personalized responses with timely and proactive interventions. We evaluate ProPer across multiple domains using a structured, gap-aware rubric that measures coverage, initiative appropriateness, and intent alignment. Our results show that ProPer improves quality scores and win rates across all domains, achieving up to 84% gains in single-turn evaluation and consistent dominance in multi-turn interactions.

Figures (6)

• Figure 1: Different agent responses to a user query. The Reactive Agent provides immediate task fulfillment without exploring user context, goals, or learning needs. The Proactive Agent clarifies task-related ambiguities to optimize the immediate solution but remains confined to the user's explicitly stated problem space. Proper, on the other hand, embodies a higher-order interaction strategy established through a learning-centric response structure.
• Figure 2: Overview of the Proper framework. During training (A), the Dimension Generating Agent (DGA) is fine-tuned on successful interactions annotated with user- and system-explicit dimensions, learning task-specific priors. At inference (B), the DGA identifies explicit and candidate implicit dimensions from the user state. A post-hoc reranker selects a calibrated subset, and the Response Generating Agent (RGA) updates the base response by selectively integrating them, balancing proactivity with user intent.
• Figure 3: Comparison of Proper with CoT prompting applied to LlaMA-8B and Qwen-8B across all datasets. Proper consistently outperforms other models even when CoT prompting enhances baseline LLMs.
• Figure 4: Performance comparison of Proper and its variants (Proper-DGA and Proper-RGA), showing the impact of removing DGA or RGA on overall results. The drop without DGA is more pronounced.
• Figure 5: Quality comparison of dimensions generated by DGA in Proper, LlaMA-8B, and Qwen-8B. Proper produces the most effective dimensions, with Qwen-8B outperforming LlaMA-8B.

Theorems & Definitions (4)

• Definition 1: Interaction Dimensions
• Definition 2: Selective Activation
• Definition 3: Post-hoc Calibrated Ranking
• Definition 4: Proper Calibration