Training Proactive and Personalized LLM Agents

TL;DR

This work targets the gap between task-focused and user-centered AI agent design by introducing UserVille, a preference-aware, interactive environment for training with diverse user personas, and PPP, a multi-objective RL framework that jointly optimizes productivity, proactivity, and personalization. Through experiments on SWE-Bench and BrowseComp-Plus, PPP yields substantial gains over baselines, demonstrates strategic, language-aware clarification behavior, and generalizes to unseen preferences and tasks. The results underscore the importance of explicitly optimizing interaction quality to build practical, user-friendly AI agents capable of adapting to real-world, underspecified user needs. Overall, the paper argues that prioritizing user-centered interaction is crucial for deploying capable and trustworthy LLM agents.

Abstract

While existing work focuses primarily on task success, we argue that effective real-world agents require optimizing three dimensions: productivity (task completion), proactivity (asking essential questions), and personalization (adapting to diverse user preferences). We introduce UserVille, an interactive environment with LLM-based user simulators enabling diverse, configurable user preferences. Leveraging UserVille, we introduce PPP, a multi-objective reinforcement learning approach that jointly optimizes all three dimensions: Productivity, Proactivity, and Personalization. Experiments on software engineering and deep research tasks show that agents trained with PPP achieve substantial improvements over strong baselines such as GPT-5 (+21.6 on average), demonstrating the ability to ask strategic clarifying questions, adapt to unseen user preferences, and improve task success through better interaction. This work demonstrates that explicitly optimizing for user-centered interaction is critical for building practical and effective AI agents.

Figures (12)

• Figure 1: Comparison of average Productivity, Proactivity, and Personalization scores on the SWE-bench Jimenez2023SWEbenchCL and BrowseComp+ Chen2025BrowseCompPlusAM datasets, where evaluation is conducted with vague user prompts rather than the original precise ones. Our proposed PPP optimization framework incentivizes high-quality agent-user interaction, achieving substantial improvements in all measured aspects. In contrast, existing LLMs (e.g., GPT-5) typically overlook interaction quality, resulting in high productivity but limited proactivity and personalization.
• Figure 2: Example of the agent interacting with the user to better understand the issue—identifying blockers and making it easy for the user to respond.
• Figure 3: UserVille simulates users with different preferences and provides feedback on interaction quality.
• Figure 4: F1 score on SWE-Bench-Verified (SWE-Func-Loc), comparing precise vs. vague initial user prompts and agents with vs. without user interaction.
• Figure 5: