Evaluating the Effectiveness and Efficiency of Demonstration Retrievers in RAG for Coding Tasks

TL;DR

This work addresses the efficiency–effectiveness trade-off in Retrieval-Augmented Generation (RAG) for coding tasks by systematically evaluating six retrievers (BM25, BM25L, SBERT_SS, ANNOY, LSH, HNSW) across three tasks (Program Synthesis, Commit Message Generation, Assertion Generation) using CodeLlama-13B. It demonstrates that BM25 is most effective on smaller knowledge bases, while approximate dense retrievers significantly improve efficiency on larger knowledge bases with only modest losses in quality. The study also shows that increasing the number of demonstrations in prompts does not reliably boost effectiveness and can increase latency or cause incorrect outputs, emphasizing careful prompt and parameter design. These findings offer practical guidance for deploying efficient and effective RAG systems in coding domains and include a replication package (ANN4RAG) to facilitate future research.

Abstract

Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge bases, achieving state-of-the-art results in various coding tasks. The core of RAG is retrieving demonstration examples, which is essential to balance effectiveness (generation quality) and efficiency (retrieval time) for optimal performance. However, the high-dimensional nature of code representations and large knowledge bases often create efficiency bottlenecks, which are overlooked in previous research. This paper systematically evaluates the efficiency-effectiveness trade-off of retrievers across three coding tasks: Program Synthesis, Commit Message Generation, and Assertion Generation. We examined six retrievers: two sparse (BM25 and BM25L) and four dense retrievers, including one exhaustive dense retriever (SBERT's Semantic Search) and three approximate dense retrievers (ANNOY, LSH, and HNSW). Our findings show that while BM25 excels in effectiveness, it suffers in efficiency as the knowledge base grows beyond 1000 entries. In large-scale retrieval, efficiency differences become more pronounced, with approximate dense retrievers offering the greatest gains. For instance, in Commit Generation task, HNSW achieves a 44x speed up, while only with a 1.74% drop in RougeL compared with BM25. Our results also show that increasing the number of demonstrations in the prompt doesn't always improve the effectiveness and can increase latency and lead to incorrect outputs. Our findings provide valuable insights for practitioners aiming to build efficient and effective RAG systems for coding tasks.

Figures (7)

• Figure 1: The general framework of RAG for coding tasks.
• Figure 2: The average tokens of prompts of different retrievers.
• Figure 3: The effectiveness of retrievers exhibits a strong correlation to the retriever's ability to recall gold demonstrations.
• Figure 4: Efficacy comparison between different retrievers with increasing knowledge base size.
• Figure 5: The trade-off between efficiency and effectiveness of retrievers. The retrievers located in up and right corner are better.