Learning Ordinal Probabilistic Reward from Preferences

TL;DR

This work tackles reward modeling for RLHF in large language models by addressing the trade-off between discriminative calibration and generative interpretability. It introduces the Ordinal Probabilistic Reward Model (OPRM), which learns a full probability distribution over ordinal quality scores and discretizes them to a finite set, enabling closed-form optimization. To align the distribution with absolute quality judgments, Region Flooding Tuning (RgFT) uses quality-level annotations to shape the score regions and preserves gradients by expanding them into a lower-triangular region, supporting semi-supervised training. Empirical results on multiple RM benchmarks show consistent accuracy gains (2.9%–7.4%), substantial calibration improvements (lower ECE), and strong data efficiency, with OPRM-RgFT achieving near-state-of-the-art performance while maintaining inference efficiency thanks to its headless, vocabulary-based scoring. Overall, OPRM with RgFT provides a principled, interpretable, and scalable framework for probabilistic reward modeling that improves both ranking fidelity and absolute quality alignment in LLM evaluation and alignment tasks.

Abstract

Reward models are crucial for aligning large language models (LLMs) with human values and intentions. Existing approaches follow either Generative (GRMs) or Discriminative (DRMs) paradigms, yet both suffer from limitations: GRMs typically demand costly point-wise supervision, while DRMs produce uncalibrated relative scores that lack probabilistic interpretation. To address these challenges, we introduce a novel reward modeling paradigm: Probabilistic Reward Model (PRM). Instead of modeling reward as a deterministic scalar, our approach treats it as a random variable, learning a full probability distribution for the quality of each response. To make this paradigm practical, we present its closed-form, discrete realization: the Ordinal Probabilistic Reward Model (OPRM), which discretizes the quality score into a finite set of ordinal ratings. Building on OPRM, we propose a data-efficient training strategy called Region Flooding Tuning (RgFT). It enables rewards to better reflect absolute text quality by incorporating quality-level annotations, which guide the model to concentrate the probability mass within corresponding rating sub-regions. Experiments on various reward model benchmarks show that our method improves accuracy by $\textbf{2.9%}\sim\textbf{7.4%}$ compared to prior reward models, demonstrating strong performance and data efficiency. Analysis of the score distribution provides evidence that our method captures not only relative rankings but also absolute quality.

Figures (5)

• Figure 1: The architectures of Ordinal Probabilistic Reward Model. Given a problem and a pair of responses, designated as chosen and rejected, the OPRM utilizes its language model (LM) head to obtain the ordinal rating probabilities for each response. A joint probability matrix is then constructed by computing the Cartesian product of these two sets of probabilities for optimization.
• Figure 2: Region Flooding Tuning. To ensure the correctness of the reward modeling, region flooding is applied to each of the three partition combinations, resulting in a lower triangular form.
• Figure 3: Comparison of score distributions for responses of high-quality (Top) and low-quality (Bottom).
• Figure 4: Ablation Study: (a) Assessing the superiority of OPRM over the BT Model. (b) Evaluating the efficacy of Weighted Average Decoding. (c) Validating the necessity of Region Flooding.
• Figure 5: Annotation Region Flooding Tuning. As annotation ambiguity increases, the target optimization region "floods" to encompass a wider set of plausible outcomes. A more uncertain annotation results in a larger target region than a more certain one.

Theorems & Definitions (6)

• Lemma B.1: Bradley-Terry Model in Sigmoid Form
• proof
• Proposition B.2
• proof
• Proposition C.1: Optimization Incentive of Preference Maximization
• proof