AP2O-Coder: Human-Inspired Progressive Optimization to Fix LLM Code Errors

TL;DR

AP2O-Coder introduces a human-inspired offline optimization framework for reducing LLM-generated code errors by building an error notebook, progressively correcting errors type-by-type, and adaptively replaying current weaknesses. It combines exam-style code generation, error diagnosis via a language-specific analyzer, progressive correction with error-frequency ordering, and adaptive quizzes to maintain focus during training. Across CodeLlama, Qwen, and DeepSeek models from 0.5B to 34B, and benchmarks EvalPlus and LiveCodeBench, AP2O-Coder achieves up to 3% improvements in pass@k while using less preference data, and shows robustness to model size and data sets. The results demonstrate reduced common errors without introducing new ones, and reveal model-size dependent error-correction dynamics (low-frequency vs high-frequency emphasis).

Abstract

LLMs' code generation capabilities have yielded substantial improvements in the effectiveness of programming tasks. However, LLM-generated code still suffers from compilation and runtime errors. Existing offline preference optimization methods primarily focus on enhancing LLMs' coding abilities using pass/fail signals in the preference data, overlooking the deep-level error types in the failed codes. To address this, we propose Adaptively Progressive Preference Optimization (AP2O) for coding (i.e., AP2O-Coder), a method that guides LLMs adaptively and methodically to reduce code errors for code generation. Specifically, we construct an error notebook from failed codes and progressively optimize the LLM to correct errors type by type. Furthermore, we adaptively replay error types to tailor to the LLM's changing weaknesses throughout the training process. Through extensive experiments on both code and general LLMs (Llama, Qwen, and DeepSeek series) with parameters ranging from 0.5B to 34B, our AP2O-Coder improves code generation performance by up to 3% in pass@k while using less preference data. Code: https://github.com/TsingZ0/AP2O

Figures (8)

• Figure 1: The illustration of our AP2O-Coder includes four steps: (1) code answer generation (exam), (2) error diagnosis (analysis), (3) progressive preference optimization (correction), and (4) adaptive error replay (quiz).
• Figure 2: The statistics of errors on the test benchmark using Qwen2.5-Coder-7B. Best viewed when zoomed in.
• Figure 3: The statistics of errors on the validation set during the quiz phase using Qwen2.5-Coder-7B. Our AP2O-Coder progressively reduces errors.
• Figure 4: The $pass@5$ and $pass@10$ on EvalPlus (HumanEval) using DeepSeek-Coder across various sizes.
• Figure 5: The preference data pair requirements for training Qwen2.5-Coder across various sizes on the MBPP training set to achieve optimal performance.