Revisit Self-Debugging with Self-Generated Tests for Code Generation

TL;DR

This work investigates self-debugging for LLM-based code generation using self-generated tests, addressing scenarios where oracle tests are unavailable. It formalizes two execution-feedback paradigms, post-execution and in-execution, and evaluates them across multiple benchmarks and models using a framework that treats an initial program $C$ and a test suite $\{(X_i,Y_i)\}_{i=1}^N$ with an executor $E$. The findings reveal that post-execution self-debugging often underperforms on basic tasks due to bias from unreliable self-generated tests, while in-execution self-debugging—guided by intermediate runtime states along basic blocks—consistently improves performance across tasks. The results highlight the value of richer runtime feedback for autonomous code refinement and suggest directions for improving test quality and the design of runtime signals to enable robust self-evolution of code-generation systems.

Abstract

Large language models (LLMs) have shown significant advancements in code generation, but still face challenges on tasks beyond their basic capabilities. Recently, the notion of self-debugging has been proposed to boost the performance of code generation by leveraging execution feedback from tests. Despite its promise, the availability of high-quality tests in real-world scenarios is limited. In this context, self-debugging with self-generated tests is a promising solution but lacks a full exploration of its limitations and practical potential. Therefore, we investigate its efficacy on diverse programming problems. To deepen our understanding, we propose two distinct paradigms for the process: post-execution and in-execution self-debugging. Within the scope of self-contained Python programming tasks, we find that post-execution self-debugging struggles on basic problems but shows potential for improvement on competitive ones, due to the bias introduced by self-generated tests. On the other hand, in-execution self-debugging enables LLMs to mitigate the bias by solely leveraging intermediate states during execution, thereby enhancing code generation.

Figures (16)

• Figure 1: Overview of self-debugging with execution feedback from self-generated tests. (1) The model generates an initial program along with a suite of tests, based on the specifications of the problem. (2) The program is executed by an executor on the self-generated tests. (3) The feedback from execution is then utilized by the model to produce a revised version of the program.
• Figure 2: The label changes when evaluating the programs with self-generated tests on HumanEval, MBPP and LiveCodeBench. True Positive (TP): correct programs pass tests; True Negative (TN): incorrect programs fail tests; False Positive (FP): incorrect programs pass tests; False Negative (FN): correct programs fail tests.
• Figure 3: An example with GPT-4o performing both post and in-execution self-debugging on a problem from HumanEval (HumanEval/135) respectively. Post-execution self-debugging wrongly corrects the program while in-execution self-debugging manages to keep the original answer.
• Figure 4: Examples of program contracts in HumanEval and MBPP. Program contracts consist of assertions that specify conditions necessary for a valid input.
• Figure 5: Code generation prompt for HumanEval.