Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners

Junhao Zheng, Xidi Cai, Qiuke Li, Duzhen Zhang, ZhongZhi Li, Yingying Zhang, Le Song, Qianli Ma

TL;DR

LifelongAgentBench introduces a unified, memory-aware benchmark for evaluating how LLM agents acquire, retain, and transfer skills across interdependent tasks in Database, Operating System, and Knowledge Graph environments. It shows that while experience replay can improve performance, it is bounded by context length and memory constraints, and introduces group self-consistency as a scalable enhancement to replay. The framework emphasizes task dependency, automatic label verification, reproducibility, and modularity, enabling fair comparisons and extensible research on memory-enabled agents. The findings highlight both the potential of lifelong learning for LLM agents and the need for memory-efficient strategies and adaptive retrieval to scale beyond short-horizon tasks.

Abstract

Lifelong learning is essential for intelligent agents operating in dynamic environments. Current large language model (LLM)-based agents, however, remain stateless and unable to accumulate or transfer knowledge over time. Existing benchmarks treat agents as static systems and fail to evaluate lifelong learning capabilities. We present LifelongAgentBench, the first unified benchmark designed to systematically assess the lifelong learning ability of LLM agents. It provides skill-grounded, interdependent tasks across three interactive environments, Database, Operating System, and Knowledge Graph, with automatic label verification, reproducibility, and modular extensibility. Extensive experiments reveal that conventional experience replay has limited effectiveness for LLM agents due to irrelevant information and context length constraints. We further introduce a group self-consistency mechanism that significantly improves lifelong learning performance. We hope LifelongAgentBench will advance the development of adaptive, memory-capable LLM agents.

LifelongAgentBench: Evaluating LLM Agents as Lifelong Learners

TL;DR

LifelongAgentBench introduces a unified, memory-aware benchmark for evaluating how LLM agents acquire, retain, and transfer skills across interdependent tasks in Database, Operating System, and Knowledge Graph environments. It shows that while experience replay can improve performance, it is bounded by context length and memory constraints, and introduces group self-consistency as a scalable enhancement to replay. The framework emphasizes task dependency, automatic label verification, reproducibility, and modularity, enabling fair comparisons and extensible research on memory-enabled agents. The findings highlight both the potential of lifelong learning for LLM agents and the need for memory-efficient strategies and adaptive retrieval to scale beyond short-horizon tasks.

Abstract

Lifelong learning is essential for intelligent agents operating in dynamic environments. Current large language model (LLM)-based agents, however, remain stateless and unable to accumulate or transfer knowledge over time. Existing benchmarks treat agents as static systems and fail to evaluate lifelong learning capabilities. We present LifelongAgentBench, the first unified benchmark designed to systematically assess the lifelong learning ability of LLM agents. It provides skill-grounded, interdependent tasks across three interactive environments, Database, Operating System, and Knowledge Graph, with automatic label verification, reproducibility, and modular extensibility. Extensive experiments reveal that conventional experience replay has limited effectiveness for LLM agents due to irrelevant information and context length constraints. We further introduce a group self-consistency mechanism that significantly improves lifelong learning performance. We hope LifelongAgentBench will advance the development of adaptive, memory-capable LLM agents.
Paper Structure (60 sections, 11 figures, 7 tables)

This paper contains 60 sections, 11 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation of Lifelong Learning for LLM Agents
  4. Data Construction
  5. Design Principles
  6. Environment Implementations
  7. Quality Control
  8. Evaluation Framework
  9. System Architecture
  10. Reproducibility and Modularity
  11. Differences from Prior Benchmarks
  12. Evaluation of LifelongAgentBench
  13. Experimental Setup
  14. Main Results
  15. Effect of Model Backbone and Task Difficulty
  16. ...and 45 more sections

Figures (11)

  • Figure 1: Overview of LifelongAgentBench: a unified dataset and evaluation framework with skill-grounded tasks, modular components, and four key properties: task dependency, label verifiability, reproducibility, and modularity.
  • Figure 2: Skill concurrency in the Database environment.
  • Figure 3: Skill distributions across tasks. Diverse and balanced skill coverage is maintained.
  • Figure 4: Illustration of group self-consistency.
  • Figure 5: The data construction process in database environment.
  • ...and 6 more figures