Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CodeRAG-Bench: Can Retrieval Augment Code Generation?

Zora Zhiruo Wang, Akari Asai, Xinyan Velocity Yu, Frank F. Xu, Yiqing Xie, Graham Neubig, Daniel Fried

TL;DR

CodeRAG-Bench introduces a unified benchmark to study retrieval-augmented code generation across diverse Python coding tasks and multiple information sources. The work systematically analyzes canonical and open retrieval setups, demonstrating that retrieved contexts can boost code quality and execution success, with GPT-4o-like models often benefiting the most. Key findings include strong gains on open-domain and repository-level tasks when libraries and solutions are retrieved, but notable challenges in retriever usefulness, context length limits, and susceptibility to distracting content. The benchmark provides a solid foundation for developing more effective RACG methods and evaluating them in realistic, code-centric scenarios.

Abstract

While language models (LMs) have proven remarkably adept at generating code, many programs are challenging for LMs to generate using their parametric knowledge alone. Providing external contexts such as library documentation can facilitate generating accurate and functional code. Despite the success of retrieval-augmented generation (RAG) in various text-oriented tasks, its potential for improving code generation remains under-explored. In this work, we conduct a systematic, large-scale analysis by asking: in what scenarios can retrieval benefit code generation models? and what challenges remain? We first curate a comprehensive evaluation benchmark, CodeRAG-Bench, encompassing three categories of code generation tasks, including basic programming, open-domain, and repository-level problems. We aggregate documents from five sources for models to retrieve contexts: competition solutions, online tutorials, library documentation, StackOverflow posts, and GitHub repositories. We examine top-performing models on CodeRAG-Bench by providing contexts retrieved from one or multiple sources. While notable gains are made in final code generation by retrieving high-quality contexts across various settings, our analysis reveals room for improvement -- current retrievers still struggle to fetch useful contexts especially with limited lexical overlap, and generators fail to improve with limited context lengths or abilities to integrate additional contexts. We hope CodeRAG-Bench serves as an effective testbed to encourage further development of advanced code-oriented RAG methods.

CodeRAG-Bench: Can Retrieval Augment Code Generation?

TL;DR

CodeRAG-Bench introduces a unified benchmark to study retrieval-augmented code generation across diverse Python coding tasks and multiple information sources. The work systematically analyzes canonical and open retrieval setups, demonstrating that retrieved contexts can boost code quality and execution success, with GPT-4o-like models often benefiting the most. Key findings include strong gains on open-domain and repository-level tasks when libraries and solutions are retrieved, but notable challenges in retriever usefulness, context length limits, and susceptibility to distracting content. The benchmark provides a solid foundation for developing more effective RACG methods and evaluating them in realistic, code-centric scenarios.

Abstract

While language models (LMs) have proven remarkably adept at generating code, many programs are challenging for LMs to generate using their parametric knowledge alone. Providing external contexts such as library documentation can facilitate generating accurate and functional code. Despite the success of retrieval-augmented generation (RAG) in various text-oriented tasks, its potential for improving code generation remains under-explored. In this work, we conduct a systematic, large-scale analysis by asking: in what scenarios can retrieval benefit code generation models? and what challenges remain? We first curate a comprehensive evaluation benchmark, CodeRAG-Bench, encompassing three categories of code generation tasks, including basic programming, open-domain, and repository-level problems. We aggregate documents from five sources for models to retrieve contexts: competition solutions, online tutorials, library documentation, StackOverflow posts, and GitHub repositories. We examine top-performing models on CodeRAG-Bench by providing contexts retrieved from one or multiple sources. While notable gains are made in final code generation by retrieving high-quality contexts across various settings, our analysis reveals room for improvement -- current retrievers still struggle to fetch useful contexts especially with limited lexical overlap, and generators fail to improve with limited context lengths or abilities to integrate additional contexts. We hope CodeRAG-Bench serves as an effective testbed to encourage further development of advanced code-oriented RAG methods.
Paper Structure (22 sections, 6 figures, 13 tables)

This paper contains 22 sections, 6 figures, 13 tables.

Table of Contents

  1. Introduction
  2. The CodeRAG-Bench
  3. Programming Problems
  4. Retrieval Sources
  5. Canonical Document Annotation
  6. Evaluation Metrics
  7. Canonical RACG
  8. Experimental Setup
  9. Retrieval Results
  10. Generation with Canonical Documents
  11. Retrieval-Augmented Code Generation
  12. RACG with Open Retrieval
  13. Related Work
  14. Conclusion
  15. Example Illustrations
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Overview of CodeRAG-Bench.
  • Figure 2: Performance with different chunking sizes.
  • Figure 3: HumanEval (left) and MBPP (right) examples with annotated canonical solutions.
  • Figure 4: DS-1000 (left) and ODEX (right) examples with annotated canonical library documentation.
  • Figure 5: RACG helps with relevant contexts (left) and hurts with distracting contexts (right).
  • ...and 1 more figures