Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning

Abstract

Reinforcement learning for code generation relies on verifiable rewards from unit test pass rates. Yet high-quality test suites are scarce, existing datasets offer limited coverage, and static rewards fail to adapt as models improve. Recent self-play methods unify code and test generation in a single model, but face a inherent dilemma: white-box access leads to self-collusion where the model produces trivial tests for easy rewards, yet black-box restriction yields generic tests that miss implementation-specific bugs. We introduce Code-A1, an adversarial co-evolution framework that jointly optimizes a Code LLM and a Test LLM with opposing objectives. The Code LLM is rewarded for passing more tests, while the Test LLM is rewarded for exposing more defects. This architectural separation eliminates self-collusion risks and safely enables white-box test generation, where the Test LLM can inspect candidate code to craft targeted adversarial tests. We further introduce a Mistake Book mechanism for experience replay and a composite reward balancing test validity with adversarial difficulty. Experiments on Qwen2.5-Coder models demonstrate that Code-A1 achieves code generation performance matching or exceeding models trained on human-annotated tests, while significantly improving test generation capability.

Figures (6)

• Figure 1: Comparison of three training paradigms for code generation. Vanilla GRPO relies on static golden tests with rigid rewards. GRPO with Self-Play unifies code and test generation but must operate in black-box mode to prevent self-collusion. Code-A1 decouples the two tasks into models with opposing objectives, safely enabling white-box test generation and stable co-evolution via the Mistake Book.
• Figure 2: Overview of the Code-A1 training framework. The Code LLM generates solutions accessible to the Test LLM for white-box testing. Generated tests are validated and merged with historical tests from the Mistake Book. The Code LLM is rewarded for passing more tests; the Test LLM is rewarded for exposing more defects, enabling curriculum-aware adversarial co-evolution.
• Figure 3: Impact of trade-off weight $\alpha$ on model capabilities using Qwen2.5-Coder-1.5B-Instruct. Left: Code generation performance (avg@32) on three benchmarks. Right: Test generation metrics on UnLeakedTestBench.
• Figure 4: Effect of Mistake Book. (a) Pass rate on newly generated tests, historical tests, and Golden Tests baseline over training. (b) Number of tests added to and removed from the Mistake Book per step.
• Figure 5: Training dynamics (Code Reward and Test Reward) over training steps for (a) Code LLM and (b) Test LLM.