Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents

Luca Gioacchini, Giuseppe Siracusano, Davide Sanvito, Kiril Gashteovski, David Friede, Roberto Bifulco, Carolin Lawrence

TL;DR

AgentQuest tackles the challenge of evaluating LLM-driven agents by introducing a modular benchmark framework with a unified driver interface and two novel metrics: progress rate and repetition rate. These metrics enable fine-grained tracking of agent advancement and action similarity, facilitating debugging and architectural improvements across diverse tasks. The framework is demonstrated on four benchmarks, including two newly added ones (Mastermind and Sudoku), and shows how metric-driven insights can guide enhancements such as memory integration to boost success rates. By making AgentQuest open-source, the authors promote extensible benchmarking, cross-benchmark comparability, and iterative improvements in generative agent architectures with practical, actionable diagnostics.

Abstract

The advances made by Large Language Models (LLMs) have led to the pursuit of LLM agents that can solve intricate, multi-step reasoning tasks. As with any research pursuit, benchmarking and evaluation are key corner stones to efficient and reliable progress. However, existing benchmarks are often narrow and simply compute overall task success. To face these issues, we propose AgentQuest -- a framework where (i) both benchmarks and metrics are modular and easily extensible through well documented and easy-to-use APIs; (ii) we offer two new evaluation metrics that can reliably track LLM agent progress while solving a task. We exemplify the utility of the metrics on two use cases wherein we identify common failure points and refine the agent architecture to obtain a significant performance increase. Together with the research community, we hope to extend AgentQuest further and therefore we make it available under https://github.com/nec-research/agentquest.

AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents

TL;DR

AgentQuest tackles the challenge of evaluating LLM-driven agents by introducing a modular benchmark framework with a unified driver interface and two novel metrics: progress rate and repetition rate. These metrics enable fine-grained tracking of agent advancement and action similarity, facilitating debugging and architectural improvements across diverse tasks. The framework is demonstrated on four benchmarks, including two newly added ones (Mastermind and Sudoku), and shows how metric-driven insights can guide enhancements such as memory integration to boost success rates. By making AgentQuest open-source, the authors promote extensible benchmarking, cross-benchmark comparability, and iterative improvements in generative agent architectures with practical, actionable diagnostics.

Abstract

The advances made by Large Language Models (LLMs) have led to the pursuit of LLM agents that can solve intricate, multi-step reasoning tasks. As with any research pursuit, benchmarking and evaluation are key corner stones to efficient and reliable progress. However, existing benchmarks are often narrow and simply compute overall task success. To face these issues, we propose AgentQuest -- a framework where (i) both benchmarks and metrics are modular and easily extensible through well documented and easy-to-use APIs; (ii) we offer two new evaluation metrics that can reliably track LLM agent progress while solving a task. We exemplify the utility of the metrics on two use cases wherein we identify common failure points and refine the agent architecture to obtain a significant performance increase. Together with the research community, we hope to extend AgentQuest further and therefore we make it available under https://github.com/nec-research/agentquest.
Paper Structure (26 sections, 4 figures, 3 tables)

This paper contains 26 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Generative AI Agents in a Nutshell
  3. Agent-Environment interaction
  4. Environment and states.
  5. Observations and actions.
  6. Benchmarking an agent
  7. AgentQuest Overview
  8. Benchmarks common interface
  9. Observations and actions.
  10. Driver.
  11. Understanding agent advancements
  12. Milestones and progress rate.
  13. Repetition rate.
  14. Adding new metrics
  15. Metric function template.
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Overview of agent-benchmark interactions in existing frameworks and in AgentQuest. AgentQuest defines a common interface to interact with the benchmarks and to compute progress metrics, easing the addition of new benchmarks and allowing researchers to evaluate and debug their agent architectures.
  • Figure 2: Average Progress rate PR$_t$ and the repetition rate RR$_t$ on Mastermind and LTP. Mastermind: It starts out with a low RR$_t$ but this increases after step 22 while the progress rate also stall at 55%. LTP: at first a higher RR$_t$ allows the agent to progress by making small variations that lead to success, but later this plateaus.
  • Figure 3: Examples of repeated actions in Mastermind and LTP. Mastermind: there is a set of unique actions at first, but then gets stuck repeating the same actions over and over. LTP: repeated actions are small variations of the same question that lead to progress.
  • Figure 4: Progress rate PR$_t$ and the repetition rate RR$_t$ on ALFWorld and Sudoku averaged over 15 runs. ALFWorld: It starts out with a low repetition rate and quick increase of the progress rate. Then a slow increase of the repetition rate enables to further increase the progress rate although less quickly. Sudoku: The progress rate quickly reaches 8%. The repetition rate then slowly increases without any positive change in the progress rate.