Planning-Driven Programming: A Large Language Model Programming Workflow

TL;DR

Planning-Driven Programming (LPW) introduces a two-phase workflow for text-to-code generation that integrates plan generation and plan verification with runtime-informed refinements. By treating the plan verification as the natural-language intended solution, LPW achieves accurate initial code and efficient bug localization, enabling more reliable debugging than prior approaches. Across seven benchmarks and multiple backbones, LPW attains state-of-the-art Pass@1 results (e.g., 98.2% on HumanEval with GPT-4o) and demonstrates model-agnostic gains. The method emphasizes the value of structured reasoning, visible tests, and iterative updates to both the solution plan and the implemented code, highlighting practical improvements for real-world code generation tasks while noting token-cost trade-offs and areas for future efficiency improvements.

Abstract

The strong performance of large language models (LLMs) raises extensive discussion on their application to code generation. Recent research suggests continuous program refinements through visible tests to improve code generation accuracy in LLMs. However, these methods suffer from LLMs' inefficiency and limited reasoning capacity. In this work, we propose an LLM programming workflow (LPW) designed to improve both initial code generation and subsequent refinements within a structured two-phase workflow. Specifically, the solution generation phase formulates a solution plan, which is then verified through visible tests to specify the intended natural language solution. Subsequently, the code implementation phase drafts an initial code according to the solution plan and its verification. If the generated code fails the visible tests, the plan verification serves as the intended solution to consistently inform the refinement process for correcting bugs. Compared to state-of-the-art methods across various existing LLMs, LPW significantly improves the Pass@1 accuracy by up to 16.4% on well-established text-to-code generation benchmarks. LPW also sets new state-of-the-art Pass@1 accuracy, achieving 98.2% on HumanEval, 84.8% on MBPP, 59.3% on LiveCode, 62.6% on APPS, and 34.7% on CodeContest, using GPT-4o as the backbone. Our code is publicly available at: https://github.com/you68681/lpw

Figures (19)

• Figure 1: The pipeline of LPW, where the components highlighted in red are exclusive to LPW.
• Figure 2: (a): An illustrated workflow of the solution generation phase in LPW. (b): Example message fragments corresponding to each workflow component for a HumanEval problem (120th) with the GPT-3.5 backbone. The detailed messages are available in Section \ref{['case_study']}.
• Figure 3: (a): An illustrated workflow of the code implementation phase in LPW. (b): Example message fragments extending from Figure \ref{['workflow_solution_generation']} and corresponding to each workflow component. See Section \ref{['case_study']} for detailes.
• Figure 4: The impact on Pass@1 accuracy with the increased number of code implementation iterations/debugging iterations on the HumanEval benchmark when leveraging GPT-3.5 as the LLM backbone. The shaded area represents the standard deviation.
• Figure 5: Pass@1 accuracy vs. average token cost per program for LDB and LPW on different benchmarks using GPT-4o as the LLM backbone. K is $10^3$. The standard deviation is too small to be visible.