DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization

TL;DR

The paper addresses how the quality of response pairs used in preference optimization affects learning outcomes, arguing that many present pairs contain noisy differences that obscure desirable signals. It introduces Distance Calibrated Reward Margin (DCRM), which combines a reward-margin proxy with distance measures to quantify the density of useful differences in a pair $(y^+,y^-)$. Empirically, higher DCRM values in training data correlate with stronger improvements on major benchmarks (AlpacaEval, MT-Bench, Arena-Hard), and three dataset types (SS-RM, DS-RM, DS-Fix) exhibit distinct DCRM distributions and learning outcomes. To exploit this, the authors propose Best-of-$N^2$ (BoN^2) pairing, selecting high-DCRM pairs from pools of candidates, which yields consistent performance gains across models and settings, and they provide a qualitative feature-analysis showing that higher-DCRM data biases learning toward desirable signals. The work offers a practical data-curation approach to enhance PO and lays a foundation for task-aligned pair selection and potential constrained decoding to target high-DCRM regions.

Abstract

Recent research has attempted to associate preference optimization (PO) performance with the underlying preference datasets. In this work, our observation is that the differences between the preferred response $y^+$ and dispreferred response $y^-$ influence what LLMs can learn, which may not match the desirable differences to learn. Therefore, we use distance and reward margin to quantify these differences, and combine them to get Distance Calibrated Reward Margin (DCRM), a metric that measures the quality of a response pair for PO. Intuitively, DCRM encourages minimal noisy differences and maximal desired differences. With this, we study 3 types of commonly used preference datasets, classified along two axes: the source of the responses and the preference labeling function. We establish a general correlation between higher DCRM of the training set and better learning outcome. Inspired by this, we propose a best-of-$N^2$ pairing method that selects response pairs with the highest DCRM. Empirically, in various settings, our method produces training datasets that can further improve models' performance on AlpacaEval, MT-Bench, and Arena-Hard over the existing training sets.

Figures (11)

• Figure 1: Top: Ideal response pairs should have fewer noisy differences (small distances) and more desired differences (large reward margins). DCRM measures response pair quality with this intuition; Bottom: Common preference datasets (SS-RM, DS-RM, DS-Fix; See § \ref{['task_setup:dataset']}) have varying locations in the distance-reward margin landscape, but none achieves an ideal combination.
• Figure 2:
• Figure 3: DCRM is positively correlated with models' performance boost on AP-L. PCC: Pearson Correlation Coefficient; Y axis: change in AP-L after training. Each point in the diagram corresponds to a trained model.
• Figure 4: Distributions of relevant (top) and irrelevant (bottom) feature differences. Each pair of adjacent blue and orange bars represents the percentage of a kind of feat. diff. ($y^+$ more helpful, $y^-$ less truthful, etc.) among the identified feat. diff. Blue: training set differences ($Y^+$-$Y^-$); Orange: differences in model outputs on AlpacaEval after or before training ($Y_{\mathrm{trained}}$-$Y_{\mathrm{ref}}$). $Y^+$-$Y^-$ and $Y_{\mathrm{trained}}$-$Y_{\mathrm{ref}}$ have similar distributions.
• Figure 5: Top: Case Study with Chat Benchmark; Bottom: Case Study with Math benchmark; Left: Example of LLM's output before training ($y_{ref}$) and after training ($y_{trained}$); Middle: Top 10 tokens whose frequency increases the most when changing from $Y^-$ to $Y^+$ in the training set; Right: Top 10 tokens whose frequency increases the most when changing from the model's output before training ($Y_{ref}$) to after training ($Y_{ref}$) on the test set.