Detecting and Correcting Hallucinations in LLM-Generated Code via Deterministic AST Analysis

TL;DR

This work tackles Knowledge Conflicting Hallucinations (KCHs) in LLM-generated code by introducing a deterministic, static-analysis pipeline that uses ASTs augmented with a dynamically introspected Knowledge Base (KB). The system deterministically detects semantic errors such as missing/invalid API usage and misused identifiers, and automatically corrects many of these issues through targeted AST edits and import insertions, all without executing the code. On a curated 200-sample Python dataset spanning five libraries, the approach achieves 100% precision in detection, 87.6% recall, a 0.934 F1-score, and corrects 77.0% of identified hallucinations, demonstrating a viable alternative to probabilistic repair methods. The results underscore the practical potential for integrating deterministic AST–KB validation into IDEs and CI pipelines to improve trust and reliability in AI-assisted code generation, with future work aimed at expanding library coverage and multi-module analysis.

Abstract

Large Language Models (LLMs) for code generation boost productivity but frequently introduce Knowledge Conflicting Hallucinations (KCHs), subtle, semantic errors, such as non-existent API parameters, that evade linters and cause runtime failures. Existing mitigations like constrained decoding or non-deterministic LLM-in-the-loop repair are often unreliable for these errors. This paper investigates whether a deterministic, static-analysis framework can reliably detect \textit{and} auto-correct KCHs. We propose a post-processing framework that parses generated code into an Abstract Syntax Tree (AST) and validates it against a dynamically-generated Knowledge Base (KB) built via library introspection. This non-executing approach uses deterministic rules to find and fix both API and identifier-level conflicts. On a manually-curated dataset of 200 Python snippets, our framework detected KCHs with 100\% precision and 87.6\% recall (0.934 F1-score), and successfully auto-corrected 77.0\% of all identified hallucinations. Our findings demonstrate that this deterministic post-processing approach is a viable and reliable alternative to probabilistic repair, offering a clear path toward trustworthy code generation.

Figures (1)

• Figure 1: Hallucination Detection and Correction Framework