IRPO: Scaling the Bradley-Terry Model via Reinforcement Learning

TL;DR

IRPO introduces a pointwise reward framework that integrates Bradley–Terry scoring into Group Relative Policy Optimization, achieving linear-time evaluation and reducing the computational burden of RLHF. By employing intergroup relative preferences and chain-of-thought reasoning, IRPO attains state-of-the-art performance among pointwise GRMs and competitive results with leading pairwise methods across multiple benchmarks. The work systematically compares pointwise and pairwise reward signals, demonstrates robust performance under various reward designs (mean, median, interval, and AUC), and highlights substantial efficiency gains, particularly in post-training settings. Overall, IRPO offers an effective, scalable alternative for RLHF reward modeling with practical implications for large-scale LLM alignment and deployment.

Abstract

Generative Reward Models (GRMs) have attracted considerable research interest in reward modeling due to their interpretability, inference-time scalability, and potential for refinement through reinforcement learning (RL). However, widely used pairwise GRMs create a computational bottleneck when integrated with RL algorithms such as Group Relative Policy Optimization (GRPO). This bottleneck arises from two factors: (i) the O(n^2) time complexity of pairwise comparisons required to obtain relative scores, and (ii) the computational overhead of repeated sampling or additional chain-of-thought (CoT) reasoning to improve performance. To address the first factor, we propose Intergroup Relative Preference Optimization (IRPO), a novel RL framework that incorporates the well-established Bradley-Terry model into GRPO. By generating a pointwise score for each response, IRPO enables efficient evaluation of arbitrarily many candidates during RL training while preserving interpretability and fine-grained reward signals. Experimental results demonstrate that IRPO achieves state-of-the-art (SOTA) performance among pointwise GRMs across multiple benchmarks, with performance comparable to that of current leading pairwise GRMs. Furthermore, we show that IRPO significantly outperforms pairwise GRMs in post-training evaluations.

Figures (4)

• Figure 1: Different reward generation paradigms and scoring patterns for reward modeling, including the Bradley-Terry model, pointwise GRMs, pairwise GRMs and listwise GRMs. We compare these modeling approaches based on the following criteria: interpretability; inference-time scalability (i.e., the ability to improve performance by allocating more computation at inference time); input flexibility (i.e., whether the method natively supports a variable number of inputs); mapping-free (whether the method must convert relative preference signals into an absolute scalar reward) and fine-grained reward capability.
• Figure 2: (a) illustrates the training of a conventional Bradley-Terry (B-T) model, while (b) illustrates the reinforcement training process for IRPO. In contrast to the B-T model, IRPO generates two sets of completions using the GRPO rollout mechanism and optimizes the model based on the relative preference between these sets. IRPO also leverages Chain-of-Thought (CoT) reasoning to enhance performance, specifically by generating a critique before assigning a score. Following the reward computation, IRPO's optimization process follows that of GRPO.
• Figure 3: Comparison of reward score variance during training under different reward design methods.
• Figure 4: Model performance on different benchmarks throughout IRPO training.