Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware

TL;DR

This paper tackles API hallucination in code generated by large language models by introducing MARIN, a framework that jointly leverages hierarchical dependency mining and dependency constrained decoding. By static-analyzing a project to extract local and global dependencies and constructing a constrained decoding process guided by an API prefix tree and parameter patterns, MARIN reduces hallucinated API usage without relying on retrieval corpora. The authors contribute APIHulBench, a real-world Java benchmark with 416 samples and two coding stages, and two metrics MiHN and MaHR to quantify hallucinations, demonstrating substantial improvements across multiple LLMs and in industrial Huawei deployments. The work advances practical, scalable mitigation of API misuse in code generation, with significant gains in exact match, edit similarity, and reduced hallucination rates, while maintaining efficient inference. Overall, MARIN offers a robust, retrieval-free solution that aligns generated APIs with project-specific constraints, enabling more reliable AI-assisted software development in real-world settings.

Abstract

Application Programming Interfaces (APIs) are crucial in modern software development. Large Language Models (LLMs) assist in automated code generation but often struggle with API hallucination, including invoking non-existent APIs and misusing existing ones in practical development scenarios. Existing studies resort to Retrieval-Augmented Generation (RAG) methods for mitigating the hallucination issue, but tend to fail since they generally ignore the structural dependencies in practical projects and do not indeed validate whether the generated APIs are available or not. To address these limitations, we propose MARIN, a framework for mitigating API hallucination in code generated by LLMs with hierarchical dependency aware. MARIN consists of two phases: Hierarchical Dependency Mining, which analyzes local and global dependencies of the current function, aiming to supplement comprehensive project context in LLMs input, and Dependency Constrained Decoding, which utilizes mined dependencies to adaptively constrain the generation process, aiming to ensure the generated APIs align with the projects specifications. To facilitate the evaluation of the degree of API hallucination, we introduce a new benchmark APIHulBench and two new metrics including Micro Hallucination Number (MiHN) and Macro Hallucination Rate (MaHR). Experiments on six state-of-the-art LLMs demonstrate that MARIN effectively reduces API hallucinations, achieving an average decrease of 67.52% in MiHN and 73.56% in MaHR compared to the RAG approach. Applied to Huaweis internal projects and two proprietary LLMs, MARIN achieves average decreases of 57.33% in MiHN and 59.41% in MaHR.

Figures (7)

• Figure 1: The motivation example: A Wrong API generated by CodeLLama-7B with RAG.
• Figure 2: Analysis of API hallucinations across different dimensions.
• Figure 3: The overview of MARIN.
• Figure 4: An example illustrating the MARIN' prompt template based on the incomplete function in Section \ref{['sec:intro']}.
• Figure 5: Evaluation results of different variants on APIHulBench-F. "-LD" denotes removing local dependency, "-GD" denotes removing global dependency and "-LG" denotes removing local and global dependencies.