An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities

TL;DR

This work empirically evaluates retrieval-augmented generation (RAF) for code generation, addressing the semantic gap between natural language and code. It analyzes three pre-trained models (CodeGen, UniXcoder, CodeT5) across CONCODE, CoNaLa, and HearthStone, and demonstrates that RAF consistently improves performance, with BM25 often being the most effective retrieval method and Sequential Integration Fusion providing a strong, cost-efficient fusion strategy. The study also extends to large language models, showing RAF can likewise boost LLM-based code generation via prompt-based retrieval cues, while highlighting trade-offs in compute costs. Practical insights include guidance on the optimal number of retrieved snippets, robust fusion strategies, and when to deploy RAF to balance performance gains against training and inference costs. Collectively, the findings offer actionable recommendations for leveraging retrieval data to enhance code generation in both small pre-trained models and larger LLMs, with implications for real-world coding assistants and AI copilots.

Abstract

Code generation aims to automatically generate code snippets of specific programming language according to natural language descriptions. The continuous advancements in deep learning, particularly pre-trained models, have empowered the code generation task to achieve remarkable performance. One main challenge of pre-trained models for code generation is the semantic gap between natural language requirements and source code. To address the issue, prior studies typically adopt a retrieval-augmented framework for the task, where the similar code snippets collected by a retrieval process can be leveraged to help understand the requirements and provide guidance for the generation process. However, there is a lack of systematic study on the application of this framework for code generation, including the impact of the final generated results and the specific usage of the framework. In this paper, we choose three popular pre-trained code models, namely CodeGen, UniXcoder, and CodeT5, to assess the impact of the quality and utilization of retrieved code on the retrieval-augmented framework. Our analysis shows that the retrieval-augmented framework is beneficial for improving the performance of the existing pre-trained models. We also provide suggestions on the utilization of the retrieval-augmented code generation framework: BM25 and Sequential Integration Fusion are recommended due to their convenience and superior performance. Sketch Filling Fusion, which extracts a sketch of relevant code, could help the model improve its performance further. Additionally, we conduct experiments to investigate the influence of the retrieval-augmented framework on large language models for code generation, showing the effectiveness of the framework, and we discuss the trade-off between performance improvement and computational costs in each phase within the framework.

Figures (4)

• Figure 1: Overview of retrieval-augmented framework for code generation.
• Figure 2: The impact of the number of retrieved code snippets using Sequential Integration Fusion on the effectiveness of CodeT5. The green line and yellow line represent the EM metric and Edit Distance metric corresponding to the left vertical axis. The red, blue and royalblue lines represent the BLEU metric, CodeBLEU metric and Simliarity of AST, respectively. All the three metrics correspond to the right vertical axis.
• Figure 3: Case study on CONCODE with retrieval-augmented framework, where the retrieval technique is BM25, fusion strategy is Sequential Integration Fusion, and the pre-trained code model is CodeT5.
• Figure 4: The most similar code snippet retrieved by different retrieval techniques on CoNaLa.