Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play

TL;DR

This paper introduces EVA, a scalable post-training framework for large language models that abandons static prompt distributions in favor of open-ended RLHF via asymmetric self-play. By framing post-training as an infinite game between a creator (prompt sampler/evolver) and a solver (response generator), EVA uses minimax-regret signals to adaptively generate informative prompts and curricula, applicable to both online and offline RLHF. The approach jointly optimizes the prompt policy and the response policy, yielding large empirical gains and enabling curricula that surpass many human-crafted prompts on challenging benchmarks like Arena-Hard. EVA demonstrates robustness across ablations, scales with reward-model quality, and promotes meaningful curricula, suggesting a new paradigm for continual RLHF and scalable alignment.

Abstract

Current reinforcement learning (RL) frameworks for large language models (LLM) post-training typically assume a fixed prompt distribution, which is sub-optimal and bottlenecks scalability. Prior works have explored prompt evolving, but are often limited to the supervised fine-tuning stage, and prompts are sampled and evolved uniformly without signals. This empirical work presents a paradigm shift: Evolving Alignment via Asymmetric Self-Play (eva), that casts post-training as an infinite game with regret-based signals for 2 players: (i) a creator, who strategically samples and creates new informative prompts and (ii) a solver, who learns to produce preferred responses. eva is the first method that allows language models to adaptively create training prompts in both offline and online RL post-training. The design is simple, easy-to-use yet remarkably effective: eva sets a new SOTA on challenging benchmarks, without any extra human prompts, e.g. it boosts the win-rate of gemma-2-9b-it on Arena-Hard by 51.6% -> 60.1% for DPO and 52.6% -> 62.4% for RLOO, surpassing claude-3-opus and catching up to gemini-1.5-pro, both of which are orders of magnitude larger. Extensive experiments show eva can create effective RL curricula and is robust across ablations. We believe adaptively evolving prompts are key to designing the next-generation RL post-training scheme.

Figures (12)

• Figure 1: Illustration on evolving alignment. Conventional RLHF is restricted to static prompt distributions. We show that it is crucial to adaptively adjust the prompt distribution during RL post-training, and further design a method allowing LLMs to robustly create new prompts with improved coverage and complexity for continual RL post-training, offering remarkable empirical gains ($\S$\ref{['sec:exp']}).
• Figure 2: Asymmetric Self-Play Pipeline: We generalize classical RLHF with open-ended RLHF, optimized with a creator-solver game. eva strategically evolves prompt distributions with a creator policy, which synthesizes prompts with a simple estimate, sample then evolve procedure; specifically, the informativeness for each prompt is elicited from reward signals. See more on our minimax-regret objective that drives the game design & different practical implementations in $\S$\ref{['sec:method']}.
• Figure 3: Illustration of gains with one round eva by DPO.
• Figure 4: eva scales with quality of reward models, under pointwise RMs with DPO (left) and pairwise RMs with SPPO (right). Note SPPO handles general preferences thus requires pairwise RMs, and DPO relies on the Bradley-Terry assumption, for which pointwise RMs are suitable.
• Figure 5: eva stays robust with more iterations.

Theorems & Definitions (1)

• Remark 1