A Systematic Literature Review of Code Hallucinations in LLMs: Characterization, Mitigation Methods, Challenges, and Future Directions for Reliable AI

TL;DR

This systematic literature review addresses code hallucinations in LLMs across code generation and other code-centric tasks. It defines code hallucination through taxonomy of factuality, faithfulness, and compatibility, and analyzes causes such as data noise, exposure bias, and semantic grounding gaps. It surveys NLP-based mitigation methods, assesses code-specific adaptations, and lists benchmarks and metrics used to evaluate fidelity. It concludes with challenges and opportunities, including unified benchmarks, explainable detection, continual learning, and multi-stage mitigation pipelines to improve reliability in code intelligence.

Abstract

Model hallucination is one of the most critical challenges faced by Large Language Models (LLMs), especially in high-stakes code intelligence tasks. As LLMs become increasingly integrated into software engineering tasks, understanding and mitigating hallucination in code becomes essential. In this survey, we provide a systematic review of hallucination phenomena in code-oriented LLMs from four key perspectives. First, we begin by surveying 60 papers to define hallucination in the context of code and summarize its primary causes, such as data noise, exposure bias, and insufficient semantic grounding, while also tracing recent trends in literature across natural language processing (NLP) and software engineering communities. Second, we review model hallucination surveys in a broader span and summarize representative hallucination mitigation strategies, such as knowledge-enhanced generation, constrained decoding, and post-editing. Third, we review approaches targeted for code intelligence and highlight code-specific challenges that aggravate hallucination, including syntax sensitivity, strict type systems, and dependence on external libraries. Meanwhile, we analyze how emerging code intelligence tasks, e.g., program analysis, symbolic execution, and unit testing, are utilized to detect and mitigate hallucinations. Fourth, we summarize current evaluation benchmarks, ranging from static metrics to dynamic checks, e.g., compilation and execution correctness, and emphasize the need for hallucination-oriented benchmarks.

Figures (14)

• Figure 1: A General Workflow of Code Intelligence. Step 1: User input; Step 2: retrieval-augmented generation incorporates external code and documentation to enrich knowledge; Step 3: model fine-tuning adapts pre-trained models to code-specific tasks and domains; Step 4: chain-of-thought reasoning enhances multi-step reasoning with structured techniques such as tree-of-thought; Step 5: iterative refinement corrects errors and repairs code; Step 6: verification results, corrections, and route back to earlier fine-tuning stages; Step 7: routed back to earlier CoT stages. and Step 8: the system delivers the final output.
• Figure 2: Three key LLM training stages.
• Figure 3: Various Knowledge Usage Techniques.
• Figure 4: An Illustrative Example of Using LLMs for Code Intelligence Tasks.
• Figure 5: Multi-perspective of Code Representation.