Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinement

Huan Zhang, Wei Cheng, Yuhan Wu, Wei Hu

TL;DR

PairCoder is proposed, a novel LLM-based framework for code generation that incorporates two collaborative LLM agents, namely a Navigator agent for high-level planning and a Driver agent for specific implementation.

Abstract

Large language models (LLMs) have achieved impressive performance on code generation. Although prior studies enhanced LLMs with prompting techniques and code refinement, they still struggle with complex programming problems due to rigid solution plans. In this paper, we draw on pair programming practices to propose PairCoder, a novel LLM-based framework for code generation. PairCoder incorporates two collaborative LLM agents, namely a Navigator agent for high-level planning and a Driver agent for specific implementation. The Navigator is responsible for proposing promising solution plans, selecting the current optimal plan, and directing the next iteration round based on execution feedback. The Driver follows the guidance of Navigator to undertake initial code generation, code testing, and refinement. This interleaved and iterative workflow involves multi-plan exploration and feedback-based refinement, which mimics the collaboration of pair programmers. We evaluate PairCoder with both open-source and closed-source LLMs on various code generation benchmarks. Extensive experimental results demonstrate the superior accuracy of PairCoder, achieving relative pass@1 improvements of 12.00%-162.43% compared to prompting LLMs directly.

A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinement

TL;DR

PairCoder is proposed, a novel LLM-based framework for code generation that incorporates two collaborative LLM agents, namely a Navigator agent for high-level planning and a Driver agent for specific implementation.

Abstract

Large language models (LLMs) have achieved impressive performance on code generation. Although prior studies enhanced LLMs with prompting techniques and code refinement, they still struggle with complex programming problems due to rigid solution plans. In this paper, we draw on pair programming practices to propose PairCoder, a novel LLM-based framework for code generation. PairCoder incorporates two collaborative LLM agents, namely a Navigator agent for high-level planning and a Driver agent for specific implementation. The Navigator is responsible for proposing promising solution plans, selecting the current optimal plan, and directing the next iteration round based on execution feedback. The Driver follows the guidance of Navigator to undertake initial code generation, code testing, and refinement. This interleaved and iterative workflow involves multi-plan exploration and feedback-based refinement, which mimics the collaboration of pair programmers. We evaluate PairCoder with both open-source and closed-source LLMs on various code generation benchmarks. Extensive experimental results demonstrate the superior accuracy of PairCoder, achieving relative pass@1 improvements of 12.00%-162.43% compared to prompting LLMs directly.
Paper Structure (30 sections, 10 figures, 7 tables, 1 algorithm)

This paper contains 30 sections, 10 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivating Example
  3. Framework
  4. Overview
  5. Navigator Agent
  6. Propose promising plans
  7. Select optimal plan
  8. Direct next iteration
  9. Driver Agent
  10. Generate initial code
  11. Perform code testing
  12. Repair buggy code
  13. Complexity analysis
  14. Experiments and Results
  15. Experiment Settings
  16. ...and 15 more sections

Figures (10)

  • Figure 1: A competitive programming problem excerpted from the CodeContest benchmark (#test87) alphacode. Both the plans and codes are generated by GPT-3.5-Turbo.
  • Figure 2: Overview of our PairCoder, in which a Navigator agent and a Driver agent collaborate on code generation.
  • Figure 3: The prompt template used by the Navigator to reflect on the given problem description and an output example of the reflection.
  • Figure 4: The prompt template for sampling multiple plans and an output example of solution plans.
  • Figure 5: The prompt template for selecting the optimal solution plan from the candidates.
  • ...and 5 more figures