Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling
Yang Cai, Weiqiang Zheng
TL;DR
The paper reframes LLM alignment as universal alignment under test-time scaling, introducing $(k,f(k))$-robust alignment and asymptotic U-alignment with rate $f(k)\to1$. It proves the optimal convergence rate $f(k)=\frac{k}{k+1}$ can be achieved with single-output policies via test-time sampling, and that no method can do better in general. It shows that existing post-training methods like NLHF and RLHF underperform for large $k$ due to loss of output diversity, and presents a multi-player alignment game whose symmetric Nash equilibrium yields the optimal rate while preserving diversity. The work also provides convergence guarantees for self-play dynamics to the MPNE and extends the framework to opponents that generate multiple responses, offering a principled path to scalable, diverse, and robust alignment in practice.
Abstract
Aligning large language models (LLMs) to serve users with heterogeneous and potentially conflicting preferences is a central challenge for personalized and trustworthy AI. We formalize an ideal notion of universal alignment through test-time scaling: for each prompt, the model produces $k\ge 1$ candidate responses and a user selects their preferred one. We introduce $(k,f(k))$-robust alignment, which requires the $k$-output model to have win rate $f(k)$ against any other single-output model, and asymptotic universal alignment (U-alignment), which requires $f(k)\to 1$ as $k\to\infty$. Our main result characterizes the optimal convergence rate: there exists a family of single-output policies whose $k$-sample product policies achieve U-alignment at rate $f(k)=\frac{k}{k+1}$, and no method can achieve a faster rate in general. We show that popular post-training methods, including Nash learning from human feedback (NLHF), can fundamentally underutilize the benefits of test-time scaling. Even though NLHF is optimal for $k=1$, sampling from the resulting (often deterministic) policy cannot guarantee win rates above $\tfrac{1}{2}$ except for an arbitrarily small slack. This stems from a lack of output diversity: existing alignment methods can collapse to a single majority-preferred response, making additional samples redundant. In contrast, our approach preserves output diversity and achieves the optimal test-time scaling rate. In particular, we propose a family of symmetric multi-player alignment games and prove that any symmetric Nash equilibrium policy of the $(k+1)$-player alignment game achieves the optimal $(k,\frac{k}{k+1})$-robust alignment. Finally, we provide theoretical convergence guarantees for self-play learning dynamics in these games and extend the framework to opponents that also generate multiple responses.