Understanding the Logic of Direct Preference Alignment through Logic

TL;DR

This paper addresses the lack of a principled framework for understanding direct preference alignment (DPA) losses used to align large language models with human preferences. It introduces a symbolic, probabilistic approach that treats model outputs as logical propositions and uses weighted model counting to define semantic losses, then develops a decompilation procedure to recover a modular preference-structure representation from any given loss. By defining Preference Structures $(\mathsf{P},\mathsf{P}_{\mathbf{C}},\mathsf{P}_{\mathbf{A}})$ and extending semantic loss to $\ell_{sl}(\overline{\mathsf{P}},\theta,D)$, the authors reveal a doubly-exponential space of definable losses ($4^{2^{n}}$ for $n$ predictions) organized into an entailment lattice. A case study demonstrates how the framework can guide the discovery of empirically competitive losses (e.g., $\ell_{cCPO}$) and shows how losses with different constraining semantics behave across datasets, offering a roadmap for systematic loss design in human-AI alignment.

Abstract

Recent direct preference alignment algorithms (DPA), such as DPO, have shown great promise in aligning large language models to human preferences. While this has motivated the development of many new variants of the original DPO loss, understanding the differences between these recent proposals, as well as developing new DPA loss functions, remains difficult given the lack of a technical and conceptual framework for reasoning about the underlying semantics of these algorithms. In this paper, we attempt to remedy this by formalizing DPA losses in terms of discrete reasoning problems. Specifically, we ask: Given an existing DPA loss, can we systematically derive a symbolic program that characterizes its semantics? We propose a novel formalism for characterizing preference losses for single model and reference model based approaches, and identify symbolic forms for a number of commonly used DPA variants. Further, we show how this formal view of preference learning sheds new light on both the size and structure of the DPA loss landscape, making it possible to not only rigorously characterize the relationships between recent loss proposals but also to systematically explore the landscape and derive new loss functions from first principles. We hope our framework and findings will help provide useful guidance to those working on human AI alignment.

Figures (10)

• Figure 1: Can we uncover the hidden logic of DPO? Here we show the decompilation of the DPO loss into a symbolic expression that expresses its high-level model behavior, along with a semantically modified version that we can compile into a novel DPO variant. We study how to translate between such loss and symbolic spaces to understand existing preference algorithms (e.g., by inspecting their semantics) and derive new algorithms from first principles (e.g., by modifying the semantics of existing approaches).
• Figure 2: What do formal representations of loss functions tell us? We show (A) two symbolic formulas related to single model preference learning with their semantics paraphrased in informal English. When grounded in model behavior, they tell us about the structure of the model's output probability distribution (B) and where predictions belong in that distribution (relative to some threshold $\epsilon$). We will later show that these formulas correspond to the losses $\ell_{\texttt{unCPO}}$ (Figure \ref{['fig:lattice']}) and the common baseline $\ell_{\texttt{CEUnl}}$ (Table \ref{['tab:comparison']}).
• Figure 3: Loss functions as truth tables. The Boolean semantics (top) of WMC and preference structures/losses: $\checkmark$ correspond to propositional models of $\mathsf{P}$, $\overline{\mathsf{P}_f}$, $\times$s to $\neg\mathsf{P}$ and $\overline{\neg\mathsf{P}_f}$, blank cells to conditioning constraints $\mathsf{P}_{\textbf{C}}$ and cells with multiple marks to $\mathsf{P}_{\textbf{A}}$. Losses (columns) are created by assigning/removing marks then counting these marks/rows $\sum$ (bottom Eq. from Eq. \ref{['eq:logistic_form_sl']}).
• Figure 4: How do we decompile losses? A visualization of our compositional decompilation procedure and main results using the example loss $\ell_{\texttt{ORPO}}$. First the original input loss (upper left) is stripped down to its core loss equation (lower left, $\log$ removed), which is then semantically translated (lower right) and mapped into a preference structure (upper right) that can be compiled back into the original loss (Thm \ref{['thm:correctness']}).
• Figure 5: What other losses are there? Here we show the loss landscape for single model preference approaches using a loss lattice showing losses (nodes) structured according to strict entailment ($\sqsubset$) and their core formulas $\mathsf{P}$ (boxes) with $\checkmark$ being the known losses. See Appendix \ref{['sec:new_losses']} for details of the individual losses and a more exhaustive lattice with DPO variants in Figure \ref{['fig:reference_lattice']}.

Theorems & Definitions (26)

• Example 1
• Example 2: reference form example
• Example 3: semantics
• Example 4: model counting and semantic loss
• Proposition 1: decompilation and standard semantic loss
• Proposition 2
• proof
• Example 5: preference structures and Boolean representations
• Proposition 3: monotonicity
• Example 6: loss entailment