Defining and Detecting the Defects of the Large Language Model-based Autonomous Agents

TL;DR

This work addresses defects that arise when LLM-driven agents blend developer-written control logic with dynamic LLM outputs, posing runtime risks such as tool invocation failures and service interruptions. It defines eight defect types by analyzing $6{,}854$ StackOverflow posts and introduces Agentable, a static-analysis tool that fuses Code Property Graphs (CPGs) with LLM-based reasoning and semantic enrichment to detect these defects. Evaluated on AgentSet ($84$ real-world Agents) and AgentTest ($78$ defect-labeled Agents), Agentable achieves an overall precision of $88.79\%$ and recall of $91.03\%$, reporting $889$ defects across $68$ hours of analysis (roughly $0.81$ hours per project). The work provides practical mitigations and a toolkit for improving the reliability of LLM-based Agents in real-world deployments, advancing understanding of code defects in agent workflows and enabling scalable detection of complex, cross-component failures.

Abstract

AI agents are systems capable of perceiving their environment, autonomously planning and executing tasks. Recent advancements in LLM have introduced a transformative paradigm for AI agents, enabling them to interact with external resources and tools through prompts. In such agents, the workflow integrates developer-written code, which manages framework construction and logic control, with LLM-generated natural language that enhances dynamic decision-making and interaction. However, discrepancies between developer-implemented logic and the dynamically generated content of LLMs in terms of behavior and expected outcomes can lead to defects, such as tool invocation failures and task execution errors. These issues introduce specific risks, leading to various defects in LLM-based AI Agents, such as service interruptions. Despite the importance of these issues, there is a lack of systematic work that focuses on analyzing LLM-based AI Agents to uncover defects in their code. In this paper, we present the first study focused on identifying and detecting defects in LLM Agents. We collected and analyzed 6,854 relevant posts from StackOverflow to define 8 types of agent defects. For each type, we provided detailed descriptions with an example. Then, we designed a static analysis tool, named Agentable, to detect the defects. Agentable leverages Code Property Graphs and LLMs to analyze Agent workflows by efficiently identifying specific code patterns and analyzing natural language descriptions. To evaluate Agentable, we constructed two datasets: AgentSet, consists of 84 real-world Agents, and AgentTest, which contains 78 Agents specifically designed to include various types of defects. Our results show that Agentable achieved an overall accuracy of 88.79% and a recall rate of 91.03%. Furthermore, our analysis reveals the 889 defects of the AgentSet, highlighting the prevalence of these defects.

Figures (12)

• Figure 1: An example of LLM-based Agent.
• Figure 2: An example of the motivation.
• Figure 3: An example of a card.
• Figure 4: An example of ADAL Defect.
• Figure 5: An example of IETI Defect.