Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation

Jiaming Shen, Ran Xu, Yennie Jun, Zhen Qin, Tianqi Liu, Carl Yang, Yi Liang, Simon Baumgartner, Michael Bendersky

TL;DR

RMBoost introduces a novel, preference-conditioned, multi-aspect synthetic data generation framework to train reward models for LLM alignment. By first generating a response, selecting a target preference label, and then producing a second response conditioned on that label and multiple evaluation aspects, RMBoost reduces labeling noise and increases data diversity. Empirical results across three datasets and multiple backbones show RMBoost outperforms existing synthetic-data baselines, with the Real+Syn data mixture delivering the strongest gains for RM performance and subsequent LLM alignment. The work also analyzes data bias, cost, and sensitivity to design choices, and discusses limitations and ethical considerations for practical deployment. Overall, RMBoost offers a scalable approach to producing high-quality synthetic preference data that enhances reward modeling and human-aligned AI systems.

Abstract

Reward models (RMs) are crucial for aligning large language models (LLMs) with human preferences. They are trained using preference datasets where each example consists of one input prompt, two responses, and a preference label. As curating a high-quality human labeled preference dataset is both time-consuming and expensive, people often rely on existing powerful LLMs for preference label generation. This can potentially introduce noise and impede RM training. In this work, we present RMBoost, a novel synthetic preference data generation paradigm to boost reward model quality. Unlike traditional methods, which generate two responses before obtaining the preference label, RMBoost first generates one response and selects a preference label, followed by generating the second more (or less) preferred response conditioned on the pre-selected preference label and the first response. This approach offers two main advantages. First, RMBoost reduces labeling noise since preference pairs are constructed intentionally. Second, RMBoost facilitates the creation of more diverse responses by incorporating various quality aspects (e.g., helpfulness, relevance, completeness) into the prompts. We conduct extensive experiments across three diverse datasets and demonstrate that RMBoost outperforms other synthetic preference data generation techniques and significantly boosts the performance of four distinct reward models.

Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation

TL;DR

RMBoost introduces a novel, preference-conditioned, multi-aspect synthetic data generation framework to train reward models for LLM alignment. By first generating a response, selecting a target preference label, and then producing a second response conditioned on that label and multiple evaluation aspects, RMBoost reduces labeling noise and increases data diversity. Empirical results across three datasets and multiple backbones show RMBoost outperforms existing synthetic-data baselines, with the Real+Syn data mixture delivering the strongest gains for RM performance and subsequent LLM alignment. The work also analyzes data bias, cost, and sensitivity to design choices, and discusses limitations and ethical considerations for practical deployment. Overall, RMBoost offers a scalable approach to producing high-quality synthetic preference data that enhances reward modeling and human-aligned AI systems.

Abstract

Reward models (RMs) are crucial for aligning large language models (LLMs) with human preferences. They are trained using preference datasets where each example consists of one input prompt, two responses, and a preference label. As curating a high-quality human labeled preference dataset is both time-consuming and expensive, people often rely on existing powerful LLMs for preference label generation. This can potentially introduce noise and impede RM training. In this work, we present RMBoost, a novel synthetic preference data generation paradigm to boost reward model quality. Unlike traditional methods, which generate two responses before obtaining the preference label, RMBoost first generates one response and selects a preference label, followed by generating the second more (or less) preferred response conditioned on the pre-selected preference label and the first response. This approach offers two main advantages. First, RMBoost reduces labeling noise since preference pairs are constructed intentionally. Second, RMBoost facilitates the creation of more diverse responses by incorporating various quality aspects (e.g., helpfulness, relevance, completeness) into the prompts. We conduct extensive experiments across three diverse datasets and demonstrate that RMBoost outperforms other synthetic preference data generation techniques and significantly boosts the performance of four distinct reward models.
Paper Structure (37 sections, 11 equations, 3 figures, 4 tables)

This paper contains 37 sections, 11 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. The RMBoost Framework
  5. Method Description
  6. Intuitions for RMBoost
  7. Analysis of RMBoost and Comparisons with Previous Approaches
  8. Experiments
  9. Experiment Settings
  10. Main Experiment Results
  11. Ablations of RMBoost
  12. Experiments with More Synthetic Data Generation Models
  13. Analysis of Generated Synthetic Data
  14. Cost and Efficiency of RMBoost
  15. Measuring RM Quality for LLM Alignment
  16. ...and 22 more sections

Figures (3)

  • Figure 1: The illustration compares RMBoost with existing paradigms for synthetic preference data generation. The two generation models can correspond to the same model (configured at non-zero temperature) that samples two different responses. (a) The RLAIF approach first generates two responses and then leverages an initial preference model (e.g., LLM with few-shot side-by-side demonstrations) to predict the preference label. (b) The RLCD method produces two responses using contrasting prompts and assigns the preference label based on the respective prompt. (c) Our approach uniquely generates the second response conditioned on the first response and a predetermined preference label.
  • Figure 2: Overview of our RMBoost framework with prompt gists. See $\S$\ref{['app:prompts']} for the complete prompts.
  • Figure 3: (a) Distributions of response re-weighting factor $\beta$. (b) Histograms of win and lose response length ratio. (c) The win rates of RMBoost over compared baselines for aligning LLMs.