LIDL: LLM Integration Defect Localization via Knowledge Graph-Enhanced Multi-Agent Analysis

TL;DR

This paper tackles defect localization in LLM‑integrated software, where failures stem from cross‑artifact interactions beyond traditional code. It introduces LIDL, a three‑agent framework that builds a code knowledge graph with LLM‑aware annotations, fuses runtime signals, LLM‑inferred hypotheses, and semantic retrieval, and uses counterfactual reasoning for validation. Evaluated on 146 real defects from 105 GitHub repos and 16 agent systems, LIDL achieves Top‑3 accuracy of $0.64$ and MAP of $0.48$, outperforming five baselines and reducing cost by $92.5\%$, demonstrating both accuracy and efficiency gains. The work highlights the importance of cross‑layer representations and semantic reasoning for LLM integration defects and provides public data and code to spur further development in reliable LLM‑driven software systems.

Abstract

LLM-integrated software, which embeds or interacts with large language models (LLMs) as functional components, exhibits probabilistic and context-dependent behaviors that fundamentally differ from those of traditional software. This shift introduces a new category of integration defects that arise not only from code errors but also from misaligned interactions among LLM-specific artifacts, including prompts, API calls, configurations, and model outputs. However, existing defect localization techniques are ineffective at identifying these LLM-specific integration defects because they fail to capture cross-layer dependencies across heterogeneous artifacts, cannot exploit incomplete or misleading error traces, and lack semantic reasoning capabilities for identifying root causes. To address these challenges, we propose LIDL, a multi-agent framework for defect localization in LLM-integrated software. LIDL (1) constructs a code knowledge graph enriched with LLM-aware annotations that represent interaction boundaries across source code, prompts, and configuration files, (2) fuses three complementary sources of error evidence inferred by LLMs to surface candidate defect locations, and (3) applies context-aware validation that uses counterfactual reasoning to distinguish true root causes from propagated symptoms. We evaluate LIDL on 146 real-world defect instances collected from 105 GitHub repositories and 16 agent-based systems. The results show that LIDL significantly outperforms five state-of-the-art baselines across all metrics, achieving a Top-3 accuracy of 0.64 and a MAP of 0.48, which represents a 64.1% improvement over the best-performing baseline. Notably, LIDL achieves these gains while reducing cost by 92.5%, demonstrating both high accuracy and cost efficiency.

Figures (9)

• Figure 1: Overview of LLM-integrated software and an example of integration defect.
• Figure 2: Representative defect cases across four LLM integration defect categories 25ninjaIssue51525autogenIssue117425camelIssue114525autogenIssue5007.
• Figure 3: The architecture of LIDL.
• Figure 4: End-to-end running example of LIDL on a real defect from gpt-researcher 25researcherIssue1027.
• Figure 5: Example of code knowledge graph with LLM annotations. The code is from an open-source LLM application 25RealCharCreate.