Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation

TL;DR

The paper introduces Sol-Ver, a self-play framework where a single LLM alternates as a solver (code generation) and verifier (unit-test generation) to iteratively produce high-quality synthetic data. Through supervised fine-tuning and direct preference optimization, Sol-Ver co-improves both capabilities, addressing data scarcity and the verifier bottleneck without external teachers. Evaluations on MBPP and LiveCodeBench using Llama 3.1 8B show meaningful relative gains in both code and test generation, with performance improving across iterations and across ablations. This approach offers a data-efficient path for robust autonomous code synthesis and verification, with potential applicability to larger models and broader coding tasks.

Abstract

Recent advances in large language models (LLMs) have improved their performance on coding benchmarks. However, improvement is plateauing due to the exhaustion of readily available high-quality data. Prior work has shown the potential of synthetic self-instruct data, but naively training on a model's own outputs can cause error accumulation, especially in coding tasks, where generalization may collapse due to overly simple or erroneous training data, highlighting the need for rigorous quality checks on synthetic data. In this work, we explore an effective approach whereby the model itself verifies the correctness of its own data. We thus propose Sol-Ver, a self-play solver-verifier framework that jointly improves a single model's code and test generation capacity. By iteratively refining code (LLM-as-a-solver) and tests (LLM-as-a-verifier) together, we boost both capabilities without relying on human annotations or larger teacher models. Experiments with the Llama 3.1 8B model demonstrate substantial performance enhancements, achieving average relative improvements of 19.63% in code generation and 17.49% in test generation on MBPP and LiveCodeBench.

Figures (5)

• Figure 1: An overview of the Sol-Ver framework. We train an LLM to both generate coding solutions (solver) and unit tests (verifier) in an iterative self-play framework, whereby synthetic preference pairs are constructed at each iteration depending on whether the code passes the generated tests or not. We show that this approach enables the model to self-improve in both capabilities (see \ref{['tab:full-performance']}).
• Figure 2: Iterative performance of our method, Sol-Ver, for test generation. Our method outperforms the baseline and SFT training for both LiveCodeBench and MBPP benchmarks, and improve across training iterations.
• Figure 3: Prompt distribution comparison with other standard coding benchmarks. We use Principal Component Analysis (PCA) for embedding dimension reduction.
• Figure 4: Pass or error distribution of synthetic data generated at each iteration.
• Figure 5: Performance of Solver-Verifier Framework.