Reliable Graph-RAG for Codebases: AST-Derived Graphs vs LLM-Extracted Knowledge Graphs

TL;DR

This study benchmarks three RAG pipelines for repository-scale code understanding: a vector-only baseline (No-Graph), an LLM-generated knowledge-graph approach (LLM-KB), and a deterministic AST-derived graph approach (DKB) built with Tree-sitter. Across three Java codebases, DKB delivers strongest multi-hop correctness with near-baseline indexing costs, while LLM-KB achieves high correctness but suffers from indexing incompleteness and substantially higher costs, and No-Graph struggles on architecture-grounding tasks. The results emphasize a practical trade-off: deterministic AST graphs provide robust, scalable grounding and lower risk of hallucinations, whereas LLM-driven graphs offer competitive accuracy at the expense of reliability and cost. The work argues for deterministic structure as a reliable backbone for code-grounded RAG in enterprise settings, while acknowledging potential hybrid approaches that blend AST-derived signals with selective LLM semantics. Overall, deterministic graph-based grounding improves coverage and reduces hallucination risk for repository-level queries with multi-hop requirements.

Abstract

Retrieval-Augmented Generation for software engineering often relies on vector similarity search, which captures topical similarity but can fail on multi-hop architectural reasoning such as controller to service to repository chains, interface-driven wiring, and inheritance. This paper benchmarks three retrieval pipelines on Java codebases (Shopizer, with additional runs on ThingsBoard and OpenMRS Core): (A) vector-only No-Graph RAG, (B) an LLM-generated knowledge graph RAG (LLM-KB), and (C) a deterministic AST-derived knowledge graph RAG (DKB) built with Tree-sitter and bidirectional traversal. Using 15 architecture and code-tracing queries per repository, we measure indexing time, query latency, corpus coverage, cost, and answer correctness. DKB builds its graph in seconds, while LLM-KB requires much longer graph generation. LLM-KB also shows indexing incompleteness: on Shopizer, 377 files are skipped or missed, reducing embedded chunk coverage and graph size compared to DKB. End-to-end cost is modest for DKB relative to the vector-only baseline but much higher for LLM-KB, especially as repository scale increases. Query latency is similar for No-Graph and DKB, while LLM-KB is slower and more variable. On the Shopizer question suite, DKB achieves the highest correctness, LLM-KB is close behind, and the vector-only baseline performs worst on upstream architectural queries and has the highest hallucination risk. Overall, deterministic AST-derived graphs provide more reliable coverage and multi-hop grounding than LLM-extracted graphs at substantially lower indexing cost.

Figures (2)

• Figure 1: Graph artifacts generated for the same Java codebase (Shopizer). DKB: nodes are project-local Java types (classes/interfaces/enums/records/annotations) discovered via Tree-sitter; edges are typed injects, extends, and implements relations. LLM-KB: nodes are LLM-emitted class_name entities; edges are depends_on relations extracted from per-file structured outputs after filtering out standard-library and framework dependencies (e.g., java.*, Spring packages). (This caption replaces the earlier placeholder request to specify node/edge definitions and filtering.)
• Figure 2: Normalized end-to-end run cost across workloads. Costs are normalized by the No-Graph baseline within each workload. LLM-KB’s cost multiplier increases substantially on the larger workload, while DKB stays near $\sim$2$\times$.