Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems

TL;DR

The paper introduces automated failure attribution for LLM multi-agent systems and presents the Who&When dataset, comprising 127 agentic systems with fine-grained annotations of the failure-responsible agent and decisive error step. It formalizes the problem, constructs both algorithm-generated and hand-crafted agentic systems, and establishes a three-method evaluation (all-at-once, step-by-step, binary search) to attribute failures. Empirical results show that even the best methods achieve only around 53.5% agent-level accuracy and 14.2% step-level accuracy, with performance sensitive to context length and ground-truth availability, underscoring the complexity of automated failure attribution. The work provides a valuable benchmark and highlights the significant need for further research, including reasoning-enhanced prompts and hybrid approaches, to enable practical debugging of multi-agent LLM systems.

Abstract

Failure attribution in LLM multi-agent systems-identifying the agent and step responsible for task failures-provides crucial clues for systems debugging but remains underexplored and labor-intensive. In this paper, we propose and formulate a new research area: automated failure attribution for LLM multi-agent systems. To support this initiative, we introduce the Who&When dataset, comprising extensive failure logs from 127 LLM multi-agent systems with fine-grained annotations linking failures to specific agents and decisive error steps. Using the Who&When, we develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons. The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps, with some methods performing below random. Even SOTA reasoning models, such as OpenAI o1 and DeepSeek R1, fail to achieve practical usability. These results highlight the task's complexity and the need for further research in this area. Code and dataset are available at https://github.com/mingyin1/Agents_Failure_Attribution

Figures (10)

• Figure 1: When developing LLMs-powered multi-agent systems, failure attribution—identifying system components responsible for task failures based on evaluation results—has received limited attention in existing research. This process is typically performed manually, demanding substantial labor and specialized expertise. In this study, we explore the potential for automating this process.
• Figure 2: Statistical analysis of the annotation process: (1) Total labor cost for annotations in human hours. (2) The proportion of uncertain annotations to total annotations during the second round. (3) Initial disagreement rates between annotators (note that we make sure to reach a consensus through a careful discussion and voting process afterwards). These results highlight the challenges involved in performing manual failure attribution.
• Figure 3: Performance comparison of three failure attribution methods on different models in both two metrics. We found the conclusion is mostly consistent with Table \ref{['tab:main_results']}.
• Figure 4: Comparison of three failure attribution methods applied to all failure logs from the hand-crafted systems in the Who&When, evaluated under varying failure log lengths across both metrics.
• Figure 5: The distances between human-annotated decisive error steps and the predicted steps for each date instance on failure logs from both algorithm-generated and hand-crafted systems.