IntentRL: Training Proactive User-intent Agents for Open-ended Deep Research via Reinforcement Learning

TL;DR

IntentRL is a framework that trains proactive agents to clarify latent user intents before starting long-horizon research, and significantly improves both intent hit rate and downstream task performance, outperforming the built-in clarify modules of closed-source DR agents and proactive LLM baselines.

Abstract

Deep Research (DR) agents extend Large Language Models (LLMs) beyond parametric knowledge by autonomously retrieving and synthesizing evidence from large web corpora into long-form reports, enabling a long-horizon agentic paradigm. However, unlike real-time conversational assistants, DR is computationally expensive and time-consuming, creating an autonomy-interaction dilemma: high autonomy on ambiguous user queries often leads to prolonged execution with unsatisfactory outcomes. To address this, we propose IntentRL, a framework that trains proactive agents to clarify latent user intents before starting long-horizon research. To overcome the scarcity of open-ended research data, we introduce a scalable pipeline that expands a few seed samples into high-quality dialogue turns via a shallow-to-deep intent refinement graph. We further adopt a two-stage reinforcement learning (RL) strategy: Stage I applies RL on offline dialogues to efficiently learn general user-interaction behavior, while Stage II uses the trained agent and a user simulator for online rollouts to strengthen adaptation to diverse user feedback. Extensive experiments show that IntentRL significantly improves both intent hit rate and downstream task performance, outperforming the built-in clarify modules of closed-source DR agents and proactive LLM baselines.

Figures (5)

• Figure 1: Scaling user interactions is more efficient to reach the user's target for open-ended deep research. The user's simple query implies a huge information set, while the underlying user intent is a subset. LLM agents will have their own implicit bias in understanding the query; without interaction, will have little overlap with the user's target. Scaling search coverage can expand the information covering but less efficient. We propose IntentRL by scaling user-agent interactions before deep research to reach better user intent alignment without increasing the deep research burdens.
• Figure 2: Demonstration of C-DAG construction. We derive a simple query from the original and extract the removed constraints as shallow intents. We then derive deep intents from the gap between the original query and the rubrics. Finally, we extend the C-DAG by adding additional intent options to question edges.
• Figure 3: The clarification quality ($\%$) and report overall performance with varying number of clarification turns on DeepResearch Bench.
• Figure 4: The performance of utilizing different training algorithms in both offline and online settings on DeepResearch Bench.
• Figure 5: Case study of our proactive agent.