When Agents Fail: A Comprehensive Study of Bugs in LLM Agents with Automated Labeling

TL;DR

The paper addresses the challenge of debugging LLM agents by conducting the first large-scale empirical study of agent bugs across Stack Overflow, GitHub, and Hugging Face, and by introducing BugReAct, an automated labeling agent. It builds a taxonomy of bug types, root causes, and effects, and analyzes which LLM agent components are most prone to faults, along with temporal trends and cross-system differences. BugReAct leverages a ReAct-style framework with external tools to annotate bugs at low cost, achieving competitive F1-scores and offering insights for developers, maintainers, and researchers. The work advances practical debugging support for agent-based software and suggests future directions in automated repair and agent-specific tooling to strengthen the reliability of LLM agents in real-world settings.

Abstract

Large Language Models (LLMs) have revolutionized intelligent application development. While standalone LLMs cannot perform any actions, LLM agents address the limitation by integrating tools. However, debugging LLM agents is difficult and costly as the field is still in it's early stage and the community is underdeveloped. To understand the bugs encountered during agent development, we present the first comprehensive study of bug types, root causes, and effects in LLM agent-based software. We collected and analyzed 1,187 bug-related posts and code snippets from Stack Overflow, GitHub, and Hugging Face forums, focused on LLM agents built with seven widely used LLM frameworks as well as custom implementations. For a deeper analysis, we have also studied the component where the bug occurred, along with the programming language and framework. This study also investigates the feasibility of automating bug identification. For that, we have built a ReAct agent named BugReAct, equipped with adequate external tools to determine whether it can detect and annotate the bugs in our dataset. According to our study, we found that BugReAct equipped with Gemini 2.5 Flash achieved a remarkable performance in annotating bug characteristics with an average cost of 0.01 USD per post/code snippet.

Figures (16)

• Figure 1: Workflow of the data-collection and labeling.
• Figure 2: Distribution of bug types across different sources.
• Figure 3: Distribution of root causes across different sources.
• Figure 4: Distribution of effects across different sources.
• Figure 5: Component distribution across bug type.