Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization

Chaoyue He, Xin Zhou, Di Wang, Hong Xu, Wei Liu, Chunyan Miao

TL;DR

This work introduces SP2DPO, a semantic per-pair generalization of Direct Preference Optimization that replaces the global temperature beta with instance-specific temperatures beta_i derived offline from structured semantic-gap annotations. The approach formalizes Alignment Process Optimization by separating offline semantic enforcement from the inner-training loop, enabling auditable data artifacts and zero online training overhead. The per-pair temperatures modulate loss curvature and saturation, not merely gradient magnitude, and are obtained via robust teacher ensembling across prompts and annotators. Empirically, SP2DPO on UltraFeedback and AlpacaEval 2.0 across four open-weight backbones achieves competitive length-controlled performance relative to tuned DPO, with JMAMP providing a robust default that reduces prompt/teacher sensitivity while avoiding per-model beta sweeps. The work highlights the benefits of data-centric control for alignment, while acknowledging limitations in judge artifacts, annotation costs, and the need for broader safety evaluations, and outlines future directions toward expert-domain alignment and online semantic curricula.

Abstract

Direct Preference Optimization (DPO) controls the trade-off between fitting preference labels and staying close to a reference model using a single global temperature beta, implicitly treating all preference pairs as equally informative. Real-world preference corpora are heterogeneous: they mix high-signal, objective failures (for example, safety, factuality, instruction violations) with low-signal or subjective distinctions (for example, style), and also include label noise. We introduce our method, SP2DPO (Semantic Per-Pair DPO), a generalization that replaces the global temperature with an instance-specific schedule beta_i pre-decided offline from structured semantic-gap annotations (category, magnitude, confidence) produced by teacher language models. We instantiate this procedure on the UltraFeedback preference corpus (59,960 pairs), enabling large-scale construction of an auditable beta_i artifact, and incur zero training-time overhead: the inner-loop optimizer remains standard DPO with beta set per pair. We focus our empirical study on AlpacaEval 2.0, reporting both raw win rate and length-controlled win rate. Across four open-weight, instruction-tuned student backbones (4B-8B), SP2DPO is competitive with a tuned global-beta DPO baseline and improves AlpacaEval 2.0 length-controlled win rate on two of four backbones, while avoiding per-model beta sweeps. All code, annotations, and artifacts will be released.

SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization

TL;DR

This work introduces SP2DPO, a semantic per-pair generalization of Direct Preference Optimization that replaces the global temperature beta with instance-specific temperatures beta_i derived offline from structured semantic-gap annotations. The approach formalizes Alignment Process Optimization by separating offline semantic enforcement from the inner-training loop, enabling auditable data artifacts and zero online training overhead. The per-pair temperatures modulate loss curvature and saturation, not merely gradient magnitude, and are obtained via robust teacher ensembling across prompts and annotators. Empirically, SP2DPO on UltraFeedback and AlpacaEval 2.0 across four open-weight backbones achieves competitive length-controlled performance relative to tuned DPO, with JMAMP providing a robust default that reduces prompt/teacher sensitivity while avoiding per-model beta sweeps. The work highlights the benefits of data-centric control for alignment, while acknowledging limitations in judge artifacts, annotation costs, and the need for broader safety evaluations, and outlines future directions toward expert-domain alignment and online semantic curricula.

Abstract

Direct Preference Optimization (DPO) controls the trade-off between fitting preference labels and staying close to a reference model using a single global temperature beta, implicitly treating all preference pairs as equally informative. Real-world preference corpora are heterogeneous: they mix high-signal, objective failures (for example, safety, factuality, instruction violations) with low-signal or subjective distinctions (for example, style), and also include label noise. We introduce our method, SP2DPO (Semantic Per-Pair DPO), a generalization that replaces the global temperature with an instance-specific schedule beta_i pre-decided offline from structured semantic-gap annotations (category, magnitude, confidence) produced by teacher language models. We instantiate this procedure on the UltraFeedback preference corpus (59,960 pairs), enabling large-scale construction of an auditable beta_i artifact, and incur zero training-time overhead: the inner-loop optimizer remains standard DPO with beta set per pair. We focus our empirical study on AlpacaEval 2.0, reporting both raw win rate and length-controlled win rate. Across four open-weight, instruction-tuned student backbones (4B-8B), SP2DPO is competitive with a tuned global-beta DPO baseline and improves AlpacaEval 2.0 length-controlled win rate on two of four backbones, while avoiding per-model beta sweeps. All code, annotations, and artifacts will be released.
Paper Structure (182 sections, 3 theorems, 59 equations, 15 figures, 16 tables, 1 algorithm)

This paper contains 182 sections, 3 theorems, 59 equations, 15 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction: From Static $\beta$ to Pre-Decided Semantic Enforcement
  2. Background and Foundations
  3. Setup and Standard DPO Objective
  4. Origin and Interpretation of $\beta$
  5. Related Work
  6. SP2DPO: Semantic Per-Pair DPO
  7. Overview
  8. Theoretical Re-framing: DPO as Semantically-Unaware Optimization
  9. SP2DPO vs. Per-Example Loss Weighting
  10. Proof.
  11. SP2DPO vs. Adaptive-Margin Methods
  12. Eliciting Semantic Temperatures via Robust Ensembling
  13. Structured semantic gap rubric (category, magnitude, confidence)
  14. Deriving the Temperature
  15. Robust Multi-Prompt Self-Ensembling
  16. ...and 167 more sections

Key Result

Proposition 1

Assume the dataset contains at least three examples whose margins $\Delta\hat{r}_i(\theta)$ attain three distinct, nonzero values on a set of parameters $\theta$ with nonzero measure. Then there exists no choice of fixed weights $\{w_i\}$ and a global temperature $\bar{\beta}$ such that unless $\beta_i$ is constant across $i$.

Figures (15)

  • Figure 1: Standard DPO vs. SP2DPO.Left: Standard DPO applies a single global temperature $\beta$ to all preference pairs, treating heterogeneous errors (e.g., factual vs. stylistic) uniformly. Right: SP2DPO assigns an instance-specific temperature $\beta_i$ via an offline semantic annotation step, yielding stronger updates for high-signal errors and conservative updates for low-signal or noisy preferences. Key shift: semantic enforcement is pre-decided as an auditable data artifact, while the training loop remains unchanged.
  • Figure 2: AE2 length-controlled win-rate changes relative to tuned DPO. Positive values indicate that semantic per-pair schedules match or exceed tuned global-$\beta$ DPO on LC; negative values indicate a gap. Values are computed from \ref{['tab:ae2_main']} and App. \ref{['tab:ae2_full']}.
  • Figure 3: Prompt token length distribution (bin width 128), with the 99th percentile and both LLM max prompt length and dataset max prompt length annotated.
  • Figure 4: Chosen vs. rejected response token length distributions (bin width 128, trimmed at 99.5th percentile).
  • Figure 5: Chosen response token length distribution (bin width 128) with the 99th percentile and dataset max response length annotated.
  • ...and 10 more figures

Theorems & Definitions (4)

  • Proposition 1: Strict non-equivalence
  • Proposition 2: Strict non-equivalence
  • proof
  • Theorem 1: Margin-sensitive generalization (informal)