ATGen: Adversarial Reinforcement Learning for Test Case Generation

TL;DR

ATGen introduces a dynamic adversarial reinforcement learning framework to automate test-case generation for LLM-written code. By training a test generator against an adversarial code generator, ATGen creates a curriculum of increasingly difficult bugs while optimizing a multi-component reward that balances output accuracy and attack success. Experiments on APPS and Codeforces show substantial gains over prompting and SFT baselines, and demonstrate practical utility as both a Best-of-N filter and a reward source for RL-based code generation. This work establishes a new paradigm where test-case generation adaptively evolves in tandem with code generators, breaking the static-difficulty ceiling of prior approaches.

Abstract

Large Language Models (LLMs) excel at code generation, yet their outputs often contain subtle bugs, for which effective test cases are a critical bottleneck. Existing test generation methods, whether based on prompting or supervised fine-tuning, rely on static datasets. This imposes a ``fixed-difficulty ceiling'', fundamentally limiting their ability to uncover novel or more complex bugs beyond their training scope. To overcome this, we introduce ATGen, a framework that trains a test case generator via adversarial reinforcement learning. ATGen pits a test generator against an adversarial code generator that continuously crafts harder bugs to evade the current policy. This dynamic loop creates a curriculum of increasing difficulty challenging current policy. The test generator is optimized via Reinforcement Learning (RL) to jointly maximize ``Output Accuracy'' and ``Attack Success'', enabling it to learn a progressively stronger policy that breaks the fixed-difficulty ceiling of static training. Extensive experiments demonstrate that ATGen significantly outperforms state-of-the-art baselines. We further validate its practical utility, showing it serves as both a more effective filter for Best-of-N inference and a higher-quality reward source for training code generation models. Our work establishes a new, dynamic paradigm for improving the reliability of LLM-generated code.

Figures (4)

• Figure 1: The overall architecture of the ATGen framework. The top panel shows the core RL training loop: the test generator (policy) receives a state ($Q, C_{\text{adver}}$) and generates a test case $T_{\text{gen}}$ (action). It then receives a multi-component reward. The bottom panel shows the adversarial data generation loop: a code generator is tasked to sample a new, harder adversarial code $C_{\text{adver}}$ that passes the current $T_{\text{gen}}$ but fails against a ground-truth test suite $T_{\text{gold}}$. This new $C_{\text{adver}}$ is then fed back into the training loop, creating a dynamic curriculum.
• Figure 2: Performance comparison of different test generation models in a Best-of-N code generation setting. ATGen (Adaptive) achieves the best performance and significantly closes the gap to the theoretical Human Expert upper bound.
• Figure 3: Final pass@1 performance of a Qwen2.5-3B-Instruct code generator after being trained via RL with rewards provided by different test generators. Using our ATGen-7B as the reward source yields a substantially stronger final code generator compared to using baseline test generators.
• Figure 4: Training curves for the code generator when using different test generators as the reward source. (a) The total reward score during the initial 50 steps of training. Using ATGen leads to a higher and more sustained reward signal. (b) The isolated code pass rate reward after 50 steps. ATGen provides a significantly more effective learning signal than UTGen.