Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance

Yaxi Lu, Shenzhi Yang, Cheng Qian, Guirong Chen, Qinyu Luo, Yesai Wu, Huadong Wang, Xin Cong, Zhong Zhang, Yankai Lin, Weiwen Liu, Yasheng Wang, Zhiyuan Liu, Fangming Liu, Maosong Sun

TL;DR

<3-5 sentence high-level summary> ProactiveAgent introduces a data-driven shift from reactive to proactive LLM-powered agents by formalizing task anticipation and initiation. The authors build ProactiveBench, a large, real-world-inspired dataset of 6,790 events across coding, writing, and daily life, and train a reward model that aligns with human judgments to automatically evaluate proactiveness. Fine-tuning open-source models on ProactiveBench yields state-of-the-art F1 scores around 66%, outperforming many baselines and closing the gap with closed-source systems. The work highlights tangible benefits for user experience through proactive assistance while candidly addressing challenges like false alarms and timing, and provides a foundation for future, privacy-conscious, autonomous agent systems.

Abstract

Agents powered by large language models have shown remarkable abilities in solving complex tasks. However, most agent systems remain reactive, limiting their effectiveness in scenarios requiring foresight and autonomous decision-making. In this paper, we tackle the challenge of developing proactive agents capable of anticipating and initiating tasks without explicit human instructions. We propose a novel data-driven approach for this problem. Firstly, we collect real-world human activities to generate proactive task predictions. These predictions are then labeled by human annotators as either accepted or rejected. The labeled data is used to train a reward model that simulates human judgment and serves as an automatic evaluator of the proactiveness of LLM agents. Building on this, we develop a comprehensive data generation pipeline to create a diverse dataset, ProactiveBench, containing 6,790 events. Finally, we demonstrate that fine-tuning models with the proposed ProactiveBench can significantly elicit the proactiveness of LLM agents. Experimental results show that our fine-tuned model achieves an F1-Score of 66.47% in proactively offering assistance, outperforming all open-source and close-source models. These results highlight the potential of our method in creating more proactive and effective agent systems, paving the way for future advancements in human-agent collaboration.

Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance

TL;DR

<3-5 sentence high-level summary> ProactiveAgent introduces a data-driven shift from reactive to proactive LLM-powered agents by formalizing task anticipation and initiation. The authors build ProactiveBench, a large, real-world-inspired dataset of 6,790 events across coding, writing, and daily life, and train a reward model that aligns with human judgments to automatically evaluate proactiveness. Fine-tuning open-source models on ProactiveBench yields state-of-the-art F1 scores around 66%, outperforming many baselines and closing the gap with closed-source systems. The work highlights tangible benefits for user experience through proactive assistance while candidly addressing challenges like false alarms and timing, and provides a foundation for future, privacy-conscious, autonomous agent systems.

Abstract

Agents powered by large language models have shown remarkable abilities in solving complex tasks. However, most agent systems remain reactive, limiting their effectiveness in scenarios requiring foresight and autonomous decision-making. In this paper, we tackle the challenge of developing proactive agents capable of anticipating and initiating tasks without explicit human instructions. We propose a novel data-driven approach for this problem. Firstly, we collect real-world human activities to generate proactive task predictions. These predictions are then labeled by human annotators as either accepted or rejected. The labeled data is used to train a reward model that simulates human judgment and serves as an automatic evaluator of the proactiveness of LLM agents. Building on this, we develop a comprehensive data generation pipeline to create a diverse dataset, ProactiveBench, containing 6,790 events. Finally, we demonstrate that fine-tuning models with the proposed ProactiveBench can significantly elicit the proactiveness of LLM agents. Experimental results show that our fine-tuned model achieves an F1-Score of 66.47% in proactively offering assistance, outperforming all open-source and close-source models. These results highlight the potential of our method in creating more proactive and effective agent systems, paving the way for future advancements in human-agent collaboration.

Paper Structure

This paper contains 45 sections, 8 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Task Definition
  5. Pipeline Overview
  6. Environment Gym
  7. Event Collection
  8. Scenario Generation
  9. Event Generation
  10. State Maintenance
  11. Proactive Agent
  12. Task Execution
  13. User Agent
  14. Experiments
  15. Reward Model Assessment
  16. ...and 30 more sections

Figures (4)

  • Figure 1: Comparison of agent systems with two types of human-agent interaction. Reactive agents passively receive user queries and then generate responses. Proactive agents infer tasks based on environmental observations and propose possible assistance requests accordingly.
  • Figure 2: Overview of the data generation process. Taking daily life as an example, this process includes modules such as the initial scenario and job setup, events generation, proactive prediction, user judgment, and action execution.
  • Figure 3: Overview of the proactive agent framework. The agent monitors new events and updates its memory to predict potential tasks.
  • Figure 4: Two types of failure: failing to detect user needs (left) and inappropriate proposing time (right). We compare the response between our fine-tuned LLaMA-3.1-Instruct-8B with other models to show the refined proactive behavior of the model.