SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning

TL;DR

SPC addresses the costly challenge of obtaining step level supervision for LLM reasoning by introducing an adversarial self play framework with a sneaky generator and a step critic that iteratively evolve via offline reinforcement learning. The approach generates labeled data without manual annotation and demonstrates strong improvements on step level verification benchmarks while also guiding test time search to boost mathematical reasoning across multiple LLMs. Empirical results show SPC surpasses strong PRMs and prompting critics on ProcessBench, DeltaBench, and PRM800K, and achieves notable gains on MATH500 and AIME2024 when used to steer reasoning. The work offers a scalable, annotation-free pathway to robust chain-of-thought reasoning in diverse open LLMs, with implications for safer and more reliable AI reasoning in real-world tasks.

Abstract

Evaluating the step-by-step reliability of large language model (LLM) reasoning, such as Chain-of-Thought, remains challenging due to the difficulty and cost of obtaining high-quality step-level supervision. In this paper, we introduce Self-Play Critic (SPC), a novel approach where a critic model evolves its ability to assess reasoning steps through adversarial self-play games, eliminating the need for manual step-level annotation. SPC involves fine-tuning two copies of a base model to play two roles, namely a "sneaky generator" that deliberately produces erroneous steps designed to be difficult to detect, and a "critic" that analyzes the correctness of reasoning steps. These two models engage in an adversarial game in which the generator aims to fool the critic, while the critic model seeks to identify the generator's errors. Using reinforcement learning based on the game outcomes, the models iteratively improve; the winner of each confrontation receives a positive reward and the loser receives a negative reward, driving continuous self-evolution. Experiments on three reasoning process benchmarks (ProcessBench, PRM800K, DeltaBench) demonstrate that our SPC progressively enhances its error detection capabilities (e.g., accuracy increases from 70.8% to 77.7% on ProcessBench) and surpasses strong baselines, including distilled R1 model. Furthermore, SPC can guide the test-time search of diverse LLMs and significantly improve their mathematical reasoning performance on MATH500 and AIME2024, surpassing those guided by state-of-the-art process reward models.

Figures (9)

• Figure 1: We continuously generate reinforcement training samples for the critic through adversarial games. The sneaky generator aims to create subtle erroneous steps to challenge the critic, while the critic must accurately distinguish between correct and incorrect steps from a mixed input of them. Benefiting from the opposing optimization objectives, both models can evolutionally learn from each other, akin to how humans improve their skills in board games through competition.
• Figure 2: The framework of our proposed SPC. We randomly select a correct step along with the partial solution before that step and feed them into the sneaky generator, which first selects one of the predefined error types and then converts the correct step into an incorrect step. The successfully generated incorrect step is then fed to the critic for error detection. If the critic successfully identifies the error, it receives a reward of +1, while the sneaky generator incurs a reward of -1. If the critic is deceived, the critic and sneaky generator are rewarded -1 and +1, respectively.
• Figure 3: Ablation study of our critic and sneaky generator. Left: The impact of different strategies on evolving critic models. Right: The success rate of sneaky generator in attacking LLM solver and its win rate against round 0 and round 1 critics.
• Figure 4: Prompt for querying GPT-4 to collect raw data of sneaky transformation CoT.
• Figure 5: Prompt for querying DeepSeek-R1-Distill-Qwen-7B to collect raw critiques with long CoT.