CodeContests-O: Powering LLMs via Feedback-Driven Iterative Test Case Generation

TL;DR

CodeContests-O tackles the bottleneck of verifiable data for reasoning LLMs by introducing a feedback-driven iterative framework that uses external execution feedback to refine test-case generators. It demonstrates that higher-quality test cases yield stronger discriminative signals and reward fidelity, leading to improved RL performance on LiveCodeBench. Empirical results show CodeContests-O achieving $TPR=89.37\%$ and $TNR=90.89\%$ across roughly $1.1\times10^7$ solutions, and RL fine-tuning with CodeContests-O yields a $+7.47\%$ gain over baselines. The work provides reproducible code and data and suggests broad applicability to other domains requiring verifiable reasoning data.

Abstract

The rise of reasoning models necessitates large-scale verifiable data, for which programming tasks serve as an ideal source. However, while competitive programming platforms provide abundant problems and solutions, high-quality test cases for verification remain scarce. Existing approaches attempt to synthesize test cases using Large Language Models (LLMs), but rely solely on the model's intrinsic generation capabilities without external feedback, frequently resulting in insufficiently diverse cases. To address this limitation, we propose a $\textbf{Feedback-Driven Iterative Framework}$ for comprehensive test case construction. Specifically, our method leverages the LLM to generate initial test cases, executes them against known correct and incorrect solutions, and utilizes the failed results as feedback to guide the LLM in refining the test cases toward high fidelity and discriminability. We then apply this method to the CodeContests dataset to construct an optimized high-quality derivative, $\textbf{CodeContests-O}$. Evaluating against the entire pool of solutions ($1.1 \times 10^7$ in total), our dataset achieves an average True Positive Rate (TPR) of $89.37\%$ and True Negative Rate (TNR) of $90.89\%$, significantly outperforming the CodeContests and CodeContests+ by margins of $4.32\%$ and $9.37\%$, respectively. Furthermore, fine-tuning the Qwen2.5-7B model on CodeContests-O results in a $9.52\%$ improvement on LiveCodeBench (Pass@1). Experiments demonstrate the effectiveness of our framework and the quality of CodeContests-O. To support reproducibility and facilitate future research, we release the $\href{https://github.com/cai-jianfeng/CodeContests-O}{code}$ and $\href{https://huggingface.co/datasets/caijanfeng/CodeContests-O}{dataset}$.

Figures (3)

• Figure 1: Pareto frontiers of TPR (True Positive Rate, proportion of correct solutions accepted, measuring test case fidelity) and TNR (True Negative Rate, proportion of incorrect solutions rejected, measuring discriminability) across different datasets. Our CodeContests-O consistently outperforms prior datasets, demonstrating superior test case quality. The iterative refinement process (iter 0-3) progressively enhances both metrics, validating the effectiveness of our feedback-driven approach.
• Figure 2: The overview of Feedback-Driven Iterative Test Case Generation. The framework begins with Initial Test Case Generation, which analyzes the problem description $P$ to produce a generator $\mathcal{G}^{(0)}$ and commands $\mathcal{C}^{(0)}$, subsequently executing them to synthesize the initial candidate test cases $\mathcal{T}^{(e, 0)}$. This is followed by a continuous loop between Execution and Feedback Collection and Feedback-Guided Refinement. In each iteration $i$, test cases $\mathcal{T}^{(e, i)}$ are evaluated against $S^{+/-}$ solutions to distill a structured feedback report $\mathcal{R}^{(i)} = \{\mathcal{F}^{(i)}, \mathcal{E}^{(i)}\}$, where $\mathcal{F}^{(i)}$ identifies false positives/negatives and $\mathcal{E}^{(i)}$ captures execution error logs. The LLM then performs root-cause analysis on $\mathcal{R}^{(i)}$ to refine the generation logic. The process terminates once quality thresholds (TPR, TNR) are satisfied or the maximum iteration $N_{max}$ is reached, ultimately yielding the CodeContests-O dataset.
• Figure 3: Effect of logic-based checker on TPR and TNR. "w/o checker" denotes string-matching evaluation, while "w/ checker" employs the logic-based checker. The checker consistently improves TPR across all datasets by correctly validating solutions with multiple valid outputs. CodeContests-O with checker achieves the optimal balance, demonstrating the effectiveness of our synchronized checker generation strategy.