RM-Distiller: Exploiting Generative LLM for Reward Model Distillation

TL;DR

RM-Distiller presents a systematic framework for distilling reward models from generative LLMs by leveraging three teacher capabilities: refinement to generate highly correlated contrastive pairs, scoring to convey precise preference strength via margin-aware objectives, and generation to regularize the RM and preserve linguistic knowledge. The approach combines contrastive refinement, margin-aware regression, self-calibrated scoring, and generative regularization within a unified objective to produce robust reward models. Empirical results across RM benchmarks and RLHF tasks show consistent improvements over traditional distillation methods, with strong data efficiency and fast domain adaptation capabilities. The work demonstrates that exploiting multifaceted teacher capabilities yields superior alignment signals and practical benefits for AI systems requiring human-aligned preferences. It also opens avenues for future work on integrating human and AI feedback in RM distillation frameworks.

Abstract

Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. Due to the difficulty of obtaining high-quality human preference annotations, distilling preferences from generative LLMs has emerged as a standard practice. However, existing approaches predominantly treat teacher models as simple binary annotators, failing to fully exploit the rich knowledge and capabilities for RM distillation. To address this, we propose RM-Distiller, a framework designed to systematically exploit the multifaceted capabilities of teacher LLMs: (1) Refinement capability, which synthesizes highly correlated response pairs to create fine-grained and contrastive signals. (2) Scoring capability, which guides the RM in capturing precise preference strength via a margin-aware optimization objective. (3) Generation capability, which incorporates the teacher's generative distribution to regularize the RM to preserve its fundamental linguistic knowledge. Extensive experiments demonstrate that RM-Distiller significantly outperforms traditional distillation methods both on RM benchmarks and reinforcement learning-based alignment, proving that exploiting multifaceted teacher capabilities is critical for effective reward modeling. To the best of our knowledge, this is the first systematic research on RM distillation from generative LLMs.

Figures (5)

• Figure 1: Compared to traditional method, RM-Distiller unlocks the multifaceted potential of teacher LLMs for better distillation.
• Figure 2: The illustration of RM-Distiller. We first initialize preference pairs from diverse candidate models, and then synthesize highly correlated response pairs. During the training phase, the student model is guided by continuous preference margins and generative regularization.
• Figure 3: A concrete example of Contrastive Refinement.
• Figure 4: The variation of statistical metrics on ShareGPT using the BT Classifier across different RL algorithms.
• Figure 5: The variation of statistical metrics on ShareGPT using the RM-Distiller across different RL algorithms.