An Empirical Study of Bugs in Modern LLM Agent Frameworks

TL;DR

This work conducts an empirical study of 998 bug reports from CrewAI and LangChain, constructing a taxonomy of 15 root causes and 7 observable symptoms across five agent lifecycle stages, showing that agent framework bugs mainly arise from'API misuse', 'API incompatibility', and'Documentation Desync'.

Abstract

LLM agents have been widely adopted in real-world applications, relying on agent frameworks for workflow execution and multi-agent coordination. As these systems scale, understanding bugs in the underlying agent frameworks becomes critical. However, existing work mainly focuses on agent-level failures, overlooking framework-level bugs. To address this gap, we conduct an empirical study of 998 bug reports from CrewAI and LangChain, constructing a taxonomy of 15 root causes and 7 observable symptoms across five agent lifecycle stages: 'Agent Initialization','Perception', 'Self-Action', 'Mutual Interaction' and 'Evolution'. Our findings show that agent framework bugs mainly arise from 'API misuse', 'API incompatibility', and 'Documentation Desync', largely concentrated in the 'Self-Action' stage. Symptoms typically appear as 'Functional Error', 'Crash', and 'Build Failure', reflecting disruptions to task progression and control flow.

Figures (3)

• Figure 1: Overview of our Three-Step Methodology
• Figure 2: Root Cause Distribution of Agent Framework Bugs
• Figure 3: Symptoms Distribution of Agent Framework Bugs