Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LiveAgentBench: Comprehensive Benchmarking of Agentic Systems Across 104 Real-World Challenges

Hao Li, Huan Wang, Jinjie Gu, Wenjie Wang, Chenyi Zhuang, Sikang Bian

TL;DR

This work presents LiveAgentBench, a comprehensive benchmark with 104 scenarios that reflect real user requirements, constructed from publicly sourced questions on social media and real-world products and uses the Social Perception-Driven Data Generation (SPDG) method.

Abstract

As large language models grow more capable, general AI agents have become increasingly prevalent in practical applications. However, existing benchmarks face significant limitations, failing to represent real-world user tasks accurately. To address this gap, we present LiveAgentBench, a comprehensive benchmark with 104 scenarios that reflect real user requirements. It is constructed from publicly sourced questions on social media and real-world products. Central to our approach is the Social Perception-Driven Data Generation (SPDG) method, a novel process we developed to ensure each question's real-world relevance, task complexity, and result verifiability. We evaluate various models, frameworks, and commercial products using LiveAgentBench, revealing their practical performance and identifying areas for improvement. This release includes 374 tasks, with 125 for validation and 249 for testing. The SPDG process enables continuous updates with fresh queries from real-world interactions.

LiveAgentBench: Comprehensive Benchmarking of Agentic Systems Across 104 Real-World Challenges

TL;DR

This work presents LiveAgentBench, a comprehensive benchmark with 104 scenarios that reflect real user requirements, constructed from publicly sourced questions on social media and real-world products and uses the Social Perception-Driven Data Generation (SPDG) method.

Abstract

As large language models grow more capable, general AI agents have become increasingly prevalent in practical applications. However, existing benchmarks face significant limitations, failing to represent real-world user tasks accurately. To address this gap, we present LiveAgentBench, a comprehensive benchmark with 104 scenarios that reflect real user requirements. It is constructed from publicly sourced questions on social media and real-world products. Central to our approach is the Social Perception-Driven Data Generation (SPDG) method, a novel process we developed to ensure each question's real-world relevance, task complexity, and result verifiability. We evaluate various models, frameworks, and commercial products using LiveAgentBench, revealing their practical performance and identifying areas for improvement. This release includes 374 tasks, with 125 for validation and 249 for testing. The SPDG process enables continuous updates with fresh queries from real-world interactions.
Paper Structure (19 sections, 3 figures, 2 tables)

This paper contains 19 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Autonomous Agents
  4. Evaluating Agents
  5. LiveAgentBench
  6. Overview
  7. Social Perception-Driven Data Generation
  8. Data Collection
  9. Data Screening
  10. Task Construction
  11. Quality Control
  12. Experiment
  13. Experimental Setup
  14. Answer extraction
  15. Result and Analysis
  16. ...and 4 more sections

Figures (3)

  • Figure 1: An overview of LiveAgentBench, introducing the construction process of the evaluation dataset from real user cases. It is accompanied by the summary results of LiveAgentBench. "W&S" represents Work and Study, "DL" represents Daily Life, "IA&P" represents Information Access and Processing, "H&SS" represents Humanities and Social Science, and "SP" represents Social Production.
  • Figure 2: 104 Real-World Challenges in LiveAgentBench.
  • Figure 3: An illustrated introduction to the SPDG process, introducing the key aspects of the SPDG process by using a specific task as an example.