The Reward Model Selection Crisis in Personalized Alignment

TL;DR

The paper tackles the deployment challenge of personalized alignment by showing that standard RM accuracy fails to predict deployment-time generation behavior under reward-guided decoding (RGD). It introduces policy accuracy as a discrimination-focused metric and Pref-LaMP, a benchmark with ground-truth user completions for direct behavioral evaluation. Across three datasets and multiple model scales, RM accuracy and policy discrimination show only weak correlation with actual generation quality, and in-context learning with retrieval (ICL-RAG) often dominates reward-guided methods at scale. The work advocates for behavioral benchmarks and practical guidance, arguing that end-to-end evaluation is essential to translate user preferences into reliable, deployment-ready generation.

Abstract

Personalized alignment from preference data has focused primarily on improving reward model (RM) accuracy, with the implicit assumption that better preference ranking translates to better personalized behavior. However, in deployment, computational constraints necessitate inference-time adaptation via reward-guided decoding (RGD) rather than per-user policy fine-tuning. This creates a critical but overlooked requirement: reward models must not only rank preferences accurately but also effectively guide token-level generation decisions. We demonstrate that standard RM accuracy fails catastrophically as a selection criterion for deployment-ready personalized alignment. Through systematic evaluation across three datasets, we introduce policy accuracy, a metric quantifying whether RGD scoring functions correctly discriminate between preferred and dispreferred responses. We show that RM accuracy correlates only weakly with this policy-level discrimination ability (Kendall's tau = 0.08--0.31). More critically, we introduce Pref-LaMP, the first personalized alignment benchmark with ground-truth user completions, enabling direct behavioral evaluation without circular reward-based metrics. On Pref-LaMP, we expose a complete decoupling between discrimination and generation: methods with 20-point RM accuracy differences produce almost identical output quality, and even methods achieving high discrimination fail to generate behaviorally aligned responses. Finally, simple in-context learning (ICL) dominates all reward-guided methods for models > 3B parameters, achieving 3-5 point ROUGE-1 gains over the best reward method at 7B scale. These findings show that the field optimizes proxy metrics that fail to predict deployment performance and do not translate preferences into real behavioral adaptation under deployment constraints.

Figures (3)

• Figure 1: RM vs Policy Accuracy Correlation on PRISM across model scales. Correlations remain consistently weak: Pearson $r$ ranges from 0.30–0.48, Spearman $\rho$ from 0.25–0.43, and Kendall $\tau$ from 0.17–0.29. While correlations slightly strengthen with scale, they remain far below what would be needed for RM accuracy to reliably predict policy performance. Notably, methods cluster vertically (similar RM accuracy, varying policy performance), demonstrating that high RM accuracy does not guarantee better policy understanding.
• Figure 2: Generation quality (ROUGE-1, ROUGE-L, BERTScore-F1) under RGD across model scales. At 0.5B-1.5B, personalized RMs marginally improve over zero-shot; at 3B-7B, ICL baselines dominate all reward-guided methods.
• Figure 3: ROUGE-1 vs. number of in-context examples. ICL-RAG consistently outperforms random ICL, with performance scaling with both model size and context length.

Theorems & Definitions (3)

• Definition 1: Reward Model Ranking Accuracy
• Definition 2: Policy Ranking Accuracy
• Definition 3: Behavioral Alignment