Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Soft Preference Optimization: Aligning Language Models to Expert Distributions

Arsalan Sharifnassab, Saber Salehkaleybar, Sina Ghiassian, Surya Kanoria, Dale Schuurmans

TL;DR

Although SPO does not require the assumption of an existing underlying reward model, it is demonstrated that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's"softness"adjustable via the softmax exponent, an algorithm parameter.

Abstract

We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying reward model, we demonstrate that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's "softness" adjustable via the softmax exponent, an algorithm parameter. We showcase SPO's methodology, its theoretical foundation, and its comparative advantages in simplicity, computational efficiency, and alignment precision.

Soft Preference Optimization: Aligning Language Models to Expert Distributions

TL;DR

Although SPO does not require the assumption of an existing underlying reward model, it is demonstrated that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's"softness"adjustable via the softmax exponent, an algorithm parameter.

Abstract

We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward model. SPO optimizes model outputs directly over a preference dataset through a natural loss function that integrates preference loss with a regularization term across the model's entire output distribution rather than limiting it to the preference dataset. Although SPO does not require the assumption of an existing underlying reward model, we demonstrate that, under the Bradley-Terry (BT) model assumption, it converges to a softmax of scaled rewards, with the distribution's "softness" adjustable via the softmax exponent, an algorithm parameter. We showcase SPO's methodology, its theoretical foundation, and its comparative advantages in simplicity, computational efficiency, and alignment precision.
Paper Structure (19 sections, 5 theorems, 56 equations, 4 tables, 1 algorithm)

This paper contains 19 sections, 5 theorems, 56 equations, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. SPO - Basic
  4. The General SPO Algorithm
  5. SPO for Other Data-Types: Best-of-$n$ Preference and Ranked Preference
  6. Comparative Analysis: SPO Versus DPO
  7. Experiments
  8. Alignment to Pairwise Preference Data
  9. Alignment to Ranking and Best-of-$n$ Preference Data
  10. Ablation Study
  11. Related Works
  12. Limitations
  13. Proof of Theorems \ref{['th:convergence']} and \ref{['th:convergence mu']}
  14. Proof of Theorem \ref{['th:convergence n']}
  15. Proof of Theorem \ref{['th:convergence ranking']}
  16. ...and 4 more sections

Key Result

Theorem 1

Suppose that the BT model holds with rewards $r(\cdot|x)$, and fix any probability distribution $\mathcal{D}$ over $\mathcal{X}\times\mathcal{Y}\times\mathcal{Y}$ that has full supportFull support in this context means that the probability distribution assigns a non-zero sampling probability to all

Theorems & Definitions (6)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 1
  • proof : Proof of Lemma \ref{['lem:app th n ineq']}