ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning

TL;DR

ResponseRank tackles the challenge of learning not just preference order but strength, by exploiting locally valid strength signals through stratified rankings and a Plackett–Luce loss with a virtual anchor. It unifies direction and magnitude learning into a single framework that reduces to Bradley–Terry in the single-comparison limit, and introduces the Pearson Distance Correlation (PDC) to quantify learned utility distances independently from ordinal accuracy. Across synthetic data, MultiPref language modeling, and RL control tasks, ResponseRank improves strength recovery (PDC) and often enhances ordinal predictions, achieving better sample efficiency and robustness to heterogeneous strength signals. The work demonstrates practical gains for RLHF and downstream decision-making, while offering a generalizable approach to incorporating imperfect strength cues like response times into reward modeling.

Abstract

Binary choices, as often used for reinforcement learning from human feedback (RLHF), convey only the direction of a preference. A person may choose apples over oranges and bananas over grapes, but which preference is stronger? Strength is crucial for decision-making under uncertainty and generalization of preference models, but hard to measure reliably. Metadata such as response times and inter-annotator agreement can serve as proxies for strength, but are often noisy and confounded. We propose ResponseRank to address the challenge of learning from noisy strength signals. Our method uses relative differences in proxy signals to rank responses to pairwise comparisons by their inferred preference strength. To control for systemic variation, we compare signals only locally within carefully constructed strata. This enables robust learning of utility differences consistent with strength-derived rankings while making minimal assumptions about the strength signal. Our contributions are threefold: (1) ResponseRank, a novel method that robustly learns preference strength by leveraging locally valid relative strength signals; (2) empirical evidence of improved sample efficiency and robustness across diverse tasks: synthetic preference learning (with simulated response times), language modeling (with annotator agreement), and RL control tasks (with simulated episode returns); and (3) the Pearson Distance Correlation (PDC), a novel metric that isolates cardinal utility learning from ordinal accuracy.

Figures (31)

• Figure 1: ResponseRank learns strength-aware preferences from rankings over comparisons derived from implicit strength signals. We start from a dataset of comparisons, consisting of a pair of objects ($a_i, b_i$), a preference ($p_i \in \{a, b\}$), a strength signal ($\tau_i$, here scalar response times), and metadata ($m_i$). (1) Comparisons are stratified using metadata $m_i$ (e.g., by annotator ID), ensuring local validity of the signal-strength relationship. The metadata can be discarded after this step. (2) Within each stratum, instances $(a_i, b_i, p_i, \tau_i)$ are normalized to $(w_i, l_i, \tau_i)$ format based on the preference label $p_i$, where $w_i$ is the preferred item and $l_i$ is the dispreferred item. (3) Rank construction then sorts these normalized tuples (e.g., by ascending response time $\tau_i$) and appends a virtual anchor element with a fixed score of 0 to create target rankings. (4) A preference predictor models these target rankings by predicting signed utility differences$s_\theta(w_i,l_i) = u_\theta(w_i) - u_\theta(l_i)$. (5) The $s_\theta$ values (and the anchor's 0) act as latent strengths for a Plackett--Luce model; NLL loss minimization yields parameters $\theta$ of the utility function (or reward model) $u_\theta$.
• Figure 2: PDC robustly isolates distance learning, unlike TCE. Heatmaps compare PDC and TCE on synthetic data under varying levels of ordinal error ($f_{\text{sign}}$, y-axis, flipping signs) and distance information loss ($f_{\text{mag}}$, x-axis, shuffling magnitudes), for original and affine-scaled ($2u + 5$) utilities. PDC remains optimal (blue) when only ordinal information is degraded ($f_{\text{mag}}=0$), decreases systematically as distance information is lost ($f_{\text{mag}}>0$), and is consistent across utility scalings and ordinal accuracy, unlike TCE which conflates these aspects.
• Figure 3: Synthetic results with RT variability across strata. Comparison of PDC (top) and accuracy (bottom) on Deterministic, Drift-Diffusion (DDM), and Stochastic datasets as a function of dataset size ($100{}$ runs; shading indicates $95\%$ CI assuming normality). BT baseline performance with the full dataset is shown as a dashed red line; ResponseRank's interpolated breakeven point is marked by an arrow. Higher PDC indicates better learned preference strength (utility distances); higher accuracy indicates better ordinal preference predictions. ResponseRank matches BT performance with only 61-83% of the data, demonstrating the efficiency of the method. The permutation baseline (ResponseRank-Perm) confirms that performance improvements stem specifically from informative strength signals rather than modeling artifacts.
• Figure 4: Response time approach comparison. We compare BT baseline (3 epochs), BT@2 (2 epochs), RR-Random (ablation: using random strength rankings of size 2), RR-RT (using 8 length buckets with size-2 constraint), RR-Stated, and RR-Agree on both MultiPref test accuracy and RewardBench 2 performance. We report mean and 95% CI (assuming normality) across 30 seeds with distinct train/test splits.
• Figure 5: Confusion matrices of the synthetic datasets.

Theorems & Definitions (7)

• Theorem 1: ResponseRank reduces to BT for a single comparison
• Definition 1: Pearson Distance Correlation
• Proposition 1: PDC Properties
• proof : Proof for \ref{['thm:responserank_bt']}
• Theorem 2: PDC Properties - Formal Statement
• proof : Outline of Proofs for \ref{['thm:pdc_properties_formal_appendix']}
• Corollary 1: PDC baseline for no distance learning