Learning Correlated Reward Models: Statistical Barriers and Opportunities

TL;DR

The paper addresses learning correlated reward models by adopting a correlated probit framework $X\sim\mathcal{N}(\mu,\Sigma)$ and showing that pairwise preference data cannot identify correlational structure due to symmetries, while best-of-three observations are both necessary and sufficient for identifiability. It develops a near-optimal polynomial-time estimator with finite-sample guarantees and proves matching lower bounds, demonstrating the statistical practicality of higher-order preference data. The authors validate their theory on synthetic and real-world datasets, including Netflix, MovieLens, and sushi preference data, showing improved modeling of correlations and personalization. Overall, the work provides a solid statistical foundation and efficient algorithms for learning correlated utilities, with clear implications for RLHF and human-aligned decision making where preferences exhibit complex dependencies.

Abstract

Random Utility Models (RUMs) are a classical framework for modeling user preferences and play a key role in reward modeling for Reinforcement Learning from Human Feedback (RLHF). However, a crucial shortcoming of many of these techniques is the Independence of Irrelevant Alternatives (IIA) assumption, which collapses \emph{all} human preferences to a universal underlying utility function, yielding a coarse approximation of the range of human preferences. On the other hand, statistical and computational guarantees for models avoiding this assumption are scarce. In this paper, we investigate the statistical and computational challenges of learning a \emph{correlated} probit model, a fundamental RUM that avoids the IIA assumption. First, we establish that the classical data collection paradigm of pairwise preference data is \emph{fundamentally insufficient} to learn correlational information, explaining the lack of statistical and computational guarantees in this setting. Next, we demonstrate that \emph{best-of-three} preference data provably overcomes these shortcomings, and devise a statistically and computationally efficient estimator with near-optimal performance. These results highlight the benefits of higher-order preference data in learning correlated utilities, allowing for more fine-grained modeling of human preferences. Finally, we validate these theoretical guarantees on several real-world datasets, demonstrating improved personalization of human preferences.

Figures (7)

• Figure 1: The probabilities $\mathop{\mathrm{\mathbb{P}}}\limits {\left\{X_i \geqslant X_j \geqslant X_k\right\}}$, for permutations, $(i,j,k)$, of $\{1,2,3\}$, correspond to the probability mass in each of the six slices of the plane denoted through .
• Figure 2: Center: ground truth covariance matrix, two figures on the right, covariance matrix learned from best-of-three observations, two figure on the left, covariance matrix learned from pairwise comparisons. We reproduce larger version these plots in Appendix \ref{['sec:impl']}.
• Figure 3: Preference of the welfare maximizing choices, left to right for $N=1,2,3$.
• Figure 4: Larger version of the first row of Figure \ref{['fig:syn']}. Top to bottom rows: pairwise, ground truth, best-of-three.
• Figure 5: Larger version of the second row of Figure \ref{['fig:syn']}. Top to bottom rows: pairwise, ground truth, best-of-three.

Theorems & Definitions (61)

• Theorem 3.2
• Theorem 4.1
• Lemma 4.2
• proof
• Lemma 4.2
• proof : Proof of \ref{['thm:mu_sigma_ident_three_items']}
• Theorem 4.3
• proof
• Theorem 5.1
• Theorem 5.2