Dual-Weighted Reinforcement Learning for Generative Preference Modeling

TL;DR

This work introduces Dual-Weighted Reinforcement Learning (DWRL), a framework that preserves the inductive bias of the Bradley–Terry preference model while enabling chain-of-thought reasoning in non-verifiable tasks. It combines a dialog-based Generative Preference Model (GPM) with two complementary weights: an instance-wise misalignment weight and a self-normalized conditional preference score, implemented via a Monte Carlo gradient estimator. The method alternates between optimizing preference scoring and thought generation, demonstrating consistent improvements over BT and GRPO baselines across HH, Instruction Following, and Math Reasoning benchmarks on Llama3 and Qwen2.5, with ablations highlighting the critical role of the misalignment weight. Overall, DWRL provides a general, reasoning-enhanced approach to learning from human preferences beyond verifiable tasks, offering practical gains and clearer interpretability through generated thoughts.

Abstract

Reinforcement learning (RL) has recently proven effective at scaling chain-of-thought (CoT) reasoning in large language models on tasks with verifiable answers. However, extending RL to more general non-verifiable tasks, typically in the format of human preference pairs, remains both challenging and underexplored. In this work, we propose Dual-Weighted Reinforcement Learning (DWRL), a new framework for preference modeling that integrates CoT reasoning with the Bradley-Terry (BT) model via a dual-weighted RL objective that preserves preference-modeling inductive bias. DWRL approximates the maximum-likelihood objective of the BT model with two complementary weights: an instance-wise misalignment weight, which emphasizes under-trained pairs misaligned with human preference, and a group-wise (self-normalized) conditional preference score, which promotes promising thoughts. In this paper, we apply DWRL to preference modeling by training generative preference models (GPMs) to first generate a thought and then predict the human preference score. Across multiple benchmarks and model scales (Llama3 and Qwen2.5), DWRL consistently outperforms both GPM baselines and scalar models, while producing coherent, interpretable thoughts. In summary, our results position DWRL as a general framework for reasoning-enhanced preference learning beyond verifiable tasks.

Figures (7)

• Figure 1: Overview of our dialog-based generative preference model (GPM), which is highlighted in the dashed box. The upper half represents the preferred response and the lower part corresponds to the less-preferred one. The green box represents the thought-generation step for judging the response. The probability of "Yes" token represents the score of a response.
• Figure 2: Overview of our proposed Dual-Weighted Reinforcement Learning algorithm (DWRL). The dashed line indicates the weighting (misalignment weight and conditional preference score) derived from the Bradley-Terry objective.
• Figure 3: Comparative results between DWRL and baselines across three preference modeling tasks: (a) helpfulness & harmlessness, (b) instruction following, and (c) math reasoning.
• Figure 4: Distribution of the number of dialog turns in the instruction-following datasets.
• Figure 5: An example sample in our instruction following dataset.