$V_0$: A Generalist Value Model for Any Policy at State Zero

TL;DR

This paper tackles the inefficiency of coupled value models in post-training RL for LLMs by introducing $V_0$, a Generalist Value Model that treats policy capability as an explicit context input $\mathcal{C}_{\pi}$. Built on a Semantic-Perception Backbone, a Residual Query Adapter, and a TabPFN probabilistic head, $V_0$ predicts $V(\mathcal{C}_{\pi}, x)$ at State Zero, enabling zero-gradient adaptation to unseen policies. An information-theoretic analysis identifies a context shortcut in naive cross-entropy training and motivates a composite objective combining a shift-invariant Pairwise Ranking Loss with soft cross-entropy to emphasize the causal interaction $I(Y; X \mid \mathcal{C})$. Empirically, $V_0$ tracks policy evolution with high stability, generalizes across distribution shifts, and yields Pareto-optimal trade-offs between performance and cost in dynamic budget allocation and inference routing. The approach offers a scalable pre-training paradigm for capability recognition, reducing the need for continual value-model updates while delivering practical gains in data efficiency and deployment cost for large-scale LLM systems.

Abstract

Policy gradient methods rely on a baseline to measure the relative advantage of an action, ensuring the model reinforces behaviors that outperform its current average capability. In the training of Large Language Models (LLMs) using Actor-Critic methods (e.g., PPO), this baseline is typically estimated by a Value Model (Critic) often as large as the policy model itself. However, as the policy continuously evolves, the value model requires expensive, synchronous incremental training to accurately track the shifting capabilities of the policy. To avoid this overhead, Group Relative Policy Optimization (GRPO) eliminates the coupled value model by using the average reward of a group of rollouts as the baseline; yet, this approach necessitates extensive sampling to maintain estimation stability. In this paper, we propose $V_0$, a Generalist Value Model capable of estimating the expected performance of any model on unseen prompts without requiring parameter updates. We reframe value estimation by treating the policy's dynamic capability as an explicit context input; specifically, we leverage a history of instruction-performance pairs to dynamically profile the model, departing from the traditional paradigm that relies on parameter fitting to perceive capability shifts. Focusing on value estimation at State Zero (i.e., the initial prompt, hence $V_0$), our model serves as a critical resource scheduler. During GRPO training, $V_0$ predicts success rates prior to rollout, allowing for efficient sampling budget allocation; during deployment, it functions as a router, dispatching instructions to the most cost-effective and suitable model. Empirical results demonstrate that $V_0$ significantly outperforms heuristic budget allocation and achieves a Pareto-optimal trade-off between performance and cost in LLM routing tasks.

Figures (7)

• Figure 1: Comparison of Training Paradigms: Traditional Value Model vs. $V_{0}$. Top: The traditional Actor-Critic paradigm (e.g., PPO) suffers from a coupling dilemma, where the value model $V^{\pi}$ requires continuous, synchronous parameter updates to track the evolving policy $\pi$. Bottom: Our proposed $V_{0}$ reframes value estimation as In-Context Learning (ICL). By treating historical instruction-performance pairs $\mathcal{C}_{\pi}$ as explicit context, $V_{0}$ perceives policy capability shifts through a single forward pass.
• Figure 2: The $V_0$ Architecture. A Semantic Backbone extracts embedding $\mathbf{h}$, which the Residual Query Adapter projects into structured features using queries $\mathbf{Q}_{\text{static}}$ and dynamic $\Delta \mathbf{Q}$. After obtaining context $\mathcal{C}_{\pi}$ and query $x$, they are fed into the TabPFN inference head.
• Figure 3: Comparison of Value Estimation Stability during Policy Training. We track the estimation performance (Intra-AUC) of $V_0$ and the Vanilla VM across the training trajectories of three different architectures. The horizontal axis is the training steps of the policy model $\pi$. While the Vanilla VM exhibits a performance lag and instability, $V_0$ maintains high, consistent accuracy from the very first step.
• Figure 4: Robust Generalization of $V_0$ Across Diverse Distribution Shifts. The grey bars represent the performance of the policy $\pi$ (left axis), while the green lines denote the AUC of $V_0$ (right axis). $V_0$ is trained solely on the source distribution (left) and directly transferred to the unseen distribution (right). Despite fluctuations in policy training stages (Early Steps vs. Late Steps), model architectures (Weak Arch. vs. Strong Arch.), or task domains (Base vs. Harder/General), $V_0$ maintains a stable and high AUC.
• Figure 5: Applications of $V_0$ in Resource Scheduling. (a) By leveraging $V_0$ for step-wise capability estimation, Deploying $V_0$ improves upon GRPO and standard budget allocation baseline without $V_0$ (see Sec. \ref{['subsec:budget_allocation']} and \ref{['eq:utility']} for more details) on OlympiadBench of Qwen3-4B-Instruct-2507. (b) $V_0$ establishes a Pareto frontier between average accuracy and inference cost across 12 benchmarks, outperforming competitive routing baselines such as EmbedLLM and Model-SAT. Please refer to \ref{['app:details_of_fig_application']} for more details.

Theorems & Definitions (14)

• Theorem 4.1
• Theorem 4.2
• Proposition 1.1
• proof
• Lemma 1.2
• proof
• Theorem 1.3
• proof
• Theorem 1.4
• proof