Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation

Jaeseok Yoo, Hojae Han, Youngwon Lee, Jaejin Kim, Seung-won Hwang

TL;DR

Two key findings are presented: retrieving examples whose presented algorithmic plans can be referenced for generating the desired behavior significantly improves generation accuracy, and converting code into pseudocode effectively captures such algorithmic plans, enhancing retrieval quality even when the source and the target PLs are different.

Abstract

Code generation with large language models has shown significant promise, especially when employing retrieval-augmented generation (RAG) with few-shot examples. However, selecting effective examples that enhance generation quality remains a challenging task, particularly when the target programming language (PL) is underrepresented. In this study, we present two key findings: (1) retrieving examples whose presented algorithmic plans can be referenced for generating the desired behavior significantly improves generation accuracy, and (2) converting code into pseudocode effectively captures such algorithmic plans, enhancing retrieval quality even when the source and the target PLs are different. Based on these findings, we propose Plan-as-query Example Retrieval for few-shot prompting in Code generation (PERC), a novel framework that utilizes algorithmic plans to identify and retrieve effective examples. We validate the effectiveness of PERC through extensive experiments on the CodeContests, HumanEval and MultiPL-E benchmarks: PERC consistently outperforms the state-of-the-art RAG methods in code generation, both when the source and target programming languages match or differ, highlighting its adaptability and robustness in diverse coding environments.

PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation

TL;DR

Two key findings are presented: retrieving examples whose presented algorithmic plans can be referenced for generating the desired behavior significantly improves generation accuracy, and converting code into pseudocode effectively captures such algorithmic plans, enhancing retrieval quality even when the source and the target PLs are different.

Abstract

Code generation with large language models has shown significant promise, especially when employing retrieval-augmented generation (RAG) with few-shot examples. However, selecting effective examples that enhance generation quality remains a challenging task, particularly when the target programming language (PL) is underrepresented. In this study, we present two key findings: (1) retrieving examples whose presented algorithmic plans can be referenced for generating the desired behavior significantly improves generation accuracy, and (2) converting code into pseudocode effectively captures such algorithmic plans, enhancing retrieval quality even when the source and the target PLs are different. Based on these findings, we propose Plan-as-query Example Retrieval for few-shot prompting in Code generation (PERC), a novel framework that utilizes algorithmic plans to identify and retrieve effective examples. We validate the effectiveness of PERC through extensive experiments on the CodeContests, HumanEval and MultiPL-E benchmarks: PERC consistently outperforms the state-of-the-art RAG methods in code generation, both when the source and target programming languages match or differ, highlighting its adaptability and robustness in diverse coding environments.

Paper Structure

This paper contains 32 sections, 6 equations, 3 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retrieval-Augmented Code Generation
  4. Utilizing Algorithmic Plans in Code Retrieval and Generation
  5. Our distinction.
  6. Preliminaries
  7. Problem-As-Query
  8. CEDAR
  9. RepoCoder
  10. PERC
  11. Plan-As-Query Example Retrieval
  12. Code Generation with Examples
  13. Experiments
  14. Experimental Setup
  15. Metrics
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Two Python and Lua code snippets have different modalities but implement the same algorithmic plans. PERC uses pseudocode describing the algorithmic plans to minimize noise from modality differences. Colors in code represent equivalent steps.
  • Figure 2: Overview of PERC: (1) plan-based retrieval and (2) code generation with few-shot prompting. PERC retrieves examples with the most similar algorithmic plans. Yellow and blue respectively signifies the triplet of the selected few-shot example and the target problem.
  • Figure 3: Source PL distribution of retrieved examples on CodeContests, under the Mixed PL Pool setting of Table \ref{['tab:cand_pls']}. The target PL, Python, is highlighted in blue.