Failure-Aware Enhancements for Large Language Model (LLM) Code Generation: An Empirical Study on Decision Framework

TL;DR

Progressive prompting significantly improves LLM-based code generation coverage over direct prompting, achieving $96.9\%$ vs $80.5\%$ with a large effect size ($d=1.63$, $p<0.001$), yet 8 of 25 projects remain incomplete. The authors introduce a failure-aware enhancement framework with three concrete pipelines—Self-Critique, Multi-Model Collaboration, and Retrieval-Augmented Generation (RAG)—and derive a data-driven decision framework that maps failure patterns to the most effective method. Their empirical study across six challenge projects shows method effectiveness hinges on failure type: Self-Critique excels on code-reviewable logic but fails on external integration, while RAG provides broad coverage and superior efficiency, with Multi-Model offering highest reliability at greater cost. The results yield actionable guidance that practitioners can apply to prioritize enhancement strategies, enabling more robust AI-assisted code generation in real-world software engineering contexts, and point to automation opportunities for failure-type diagnosis and method selection in future work.

Abstract

Large language models (LLMs) show promise for automating software development by translating requirements into code. However, even advanced prompting workflows like progressive prompting often leave some requirements unmet. Although methods such as self-critique, multi-model collaboration, and retrieval-augmented generation (RAG) have been proposed to address these gaps, developers lack clear guidance on when to use each. In an empirical study of 25 GitHub projects, we found that progressive prompting achieves 96.9% average task completion, significantly outperforming direct prompting (80.5%, Cohen's d=1.63, p<0.001) but still leaving 8 projects incomplete. For 6 of the most representative projects, we evaluated each enhancement strategy across 4 failure types. Our results reveal that method effectiveness depends critically on failure characteristics: Self-Critique succeeds on code-reviewable logic errors but fails completely on external service integration (0% improvement), while RAG achieves highest completion across all failure types with superior efficiency. Based on these findings, we propose a decision framework that maps each failure pattern to the most suitable enhancement method, giving practitioners practical, data-driven guidance instead of trial-and-error.

Figures (1)

• Figure 1: Time vs Improvement Efficiency. RAG-Assisted (green squares); Self-Critique (red circles); Multi-Model (blue triangles)