Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalizing while preserving monotonicity in comparison-based preference learning models

Julien Fageot, Peva Blanchard, Gilles Bareilles, Lê-Nguyên Hoang

TL;DR

This work investigates monotonicity guarantees in comparison-based preference learning and identifies that many existing models fail to preserve a previously preferred item's score after updates. It introduces Linear Generalized Bradley-Terry with Diffusion Priors, which combines linear embeddings and a prior similarity structure to generalize to uncompared alternatives while preserving monotonicity under suitable embeddings. A key contribution is the notion of good embeddings and the proof that diffusion embeddings (including one-hot encodings) are good, yielding monotonicity. Empirically, diffusion-prior models achieve higher accuracy with limited data and show benefits on real-world data, such as YouTube video rankings, indicating practical impact for trustworthy preference learning systems.

Abstract

If you tell a learning model that you prefer an alternative $a$ over another alternative $b$, then you probably expect the model to be monotone, that is, the valuation of $a$ increases, and that of $b$ decreases. Yet, perhaps surprisingly, many widely deployed comparison-based preference learning models, including large language models, fail to have this guarantee. Until now, the only comparison-based preference learning algorithms that were proved to be monotone are the Generalized Bradley-Terry models. Yet, these models are unable to generalize to uncompared data. In this paper, we advance the understanding of the set of models with generalization ability that are monotone. Namely, we propose a new class of Linear Generalized Bradley-Terry models with Diffusion Priors, and identify sufficient conditions on alternatives' embeddings that guarantee monotonicity. Our experiments show that this monotonicity is far from being a general guarantee, and that our new class of generalizing models improves accuracy, especially when the dataset is limited.

Generalizing while preserving monotonicity in comparison-based preference learning models

TL;DR

This work investigates monotonicity guarantees in comparison-based preference learning and identifies that many existing models fail to preserve a previously preferred item's score after updates. It introduces Linear Generalized Bradley-Terry with Diffusion Priors, which combines linear embeddings and a prior similarity structure to generalize to uncompared alternatives while preserving monotonicity under suitable embeddings. A key contribution is the notion of good embeddings and the proof that diffusion embeddings (including one-hot encodings) are good, yielding monotonicity. Empirically, diffusion-prior models achieve higher accuracy with limited data and show benefits on real-world data, such as YouTube video rankings, indicating practical impact for trustworthy preference learning systems.

Abstract

If you tell a learning model that you prefer an alternative over another alternative , then you probably expect the model to be monotone, that is, the valuation of increases, and that of decreases. Yet, perhaps surprisingly, many widely deployed comparison-based preference learning models, including large language models, fail to have this guarantee. Until now, the only comparison-based preference learning algorithms that were proved to be monotone are the Generalized Bradley-Terry models. Yet, these models are unable to generalize to uncompared data. In this paper, we advance the understanding of the set of models with generalization ability that are monotone. Namely, we propose a new class of Linear Generalized Bradley-Terry models with Diffusion Priors, and identify sufficient conditions on alternatives' embeddings that guarantee monotonicity. Our experiments show that this monotonicity is far from being a general guarantee, and that our new class of generalizing models improves accuracy, especially when the dataset is limited.

Paper Structure

This paper contains 34 sections, 14 theorems, 50 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related works.
  4. Paper structure.
  5. Monotonicity of Scoring Models
  6. Notations and operations on datasets
  7. Exchange.
  8. Shuffle.
  9. Append.
  10. Update.
  11. Monotonicity
  12. (Generalized) Bradley-Terry and monotonicity
  13. Linear GBT with Diffusion Prior
  14. Learning with prior similarities
  15. Monotonicity and diffusion
  16. ...and 19 more sections

Key Result

Proposition 1

For any root law $f$, and $\sigma > 0$, $\textsc{GBT}_{f, \sigma}$ is monotone.Consider a root law $f$, a scalar $\sigma>0$, and two datasets ${\bf D}$, ${\bf D}'$ which contains at most one comparison between any pair $(a, b)\in\mathcal{A}^2$. Then, for all $a$, ${\bf D} \succeq_a{\bf D}'$ implies

Figures (3)

  • Figure 1: Left pane: Probability that a Gaussian i.i.d embedding $x$ is a good embedding for $2 \leq A \leq 15$ and $1 \leq D \leq 15$. Right pane: As for the left pane with embedding $Ix^T$.
  • Figure 2: Left pane: nMSE as a function of $D$ for $A = 25$ alternatives and $N = 500$ comparisons over $100$ seeds. Blue curve with $Ix^T$ (full embedding), orange curve with embedding $I$ (classical GBT), and green curve with embedding $x$ (features only). Right pane: nMSE with respect to the number of comparisons $N$ for $A = 20$, $D = 10$, and $1000$ seeds. Blue curve: GBT with one-hot encoding; Orange curve: GBT. Every curve is displayed with its error bar (using $1.96 \sigma / \sqrt{\text{n\_seeds}}$).
  • Figure 3: Validation risks of $\textsc{GBT}_{f,1,x,0}$ (with embeddings, on the left) and $\textsc{GBT}_{f,1,I_N,0}$ (without embeddings, on the right), estimated using $10$-fold cross validation. The box plots report the minimum, 1st quartile, median, 3rd quartile and maximum. Outliers are also shown.

Theorems & Definitions (35)

  • Definition 1: Favoring $a$
  • Definition 2: Monotonicity
  • Remark 1
  • Proposition 1: Th. 2, DBLP:conf/aaai/FageotFHV24
  • Definition 3: Linear GBT with Diffusion Prior
  • Proposition 2
  • proof
  • Definition 4: Super-Laplacian matrix
  • Definition 5: Diffusion embedding
  • Theorem 1: Monotonicity with diffusion embeddings
  • ...and 25 more