Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes

Zhuocheng Gong, Jian Guan, Wei Wu, Huishuai Zhang, Dongyan Zhao

TL;DR

Latent Preference Coding (LPC) introduces a discrete latent variable framework for aligning LLMs to human preferences by modeling multiple underlying factors with a codebook of latent codes. Using a prior $p(z|x)$ and a posterior $q(z|x,y_w\succ y_l)$, LPC integrates with offline RLHF objectives (e.g., $\text{DPO}$, $\text{SimPO}$, $\text{IPO}$) via a variational loss that combines likelihood over preferred pairs with a KL penalty. Empirical results across multiple base models and benchmarks show LPC improves downstream task performance and preference accuracy, and analyses reveal its latent codes capture diverse preference signals and offer robustness to noisy data. The approach provides a versatile, generalizable path toward more robust and nuanced alignment of powerful LLMs across tasks and populations.

Abstract

Large language models (LLMs) have achieved remarkable success, yet aligning their generations with human preferences remains a critical challenge. Existing approaches to preference modeling often rely on an explicit or implicit reward function, overlooking the intricate and multifaceted nature of human preferences that may encompass conflicting factors across diverse tasks and populations. To address this limitation, we introduce Latent Preference Coding (LPC), a novel framework that models the implicit factors as well as their combinations behind holistic preferences using discrete latent codes. LPC seamlessly integrates with various offline alignment algorithms, automatically inferring the underlying factors and their importance from data without relying on pre-defined reward functions and hand-crafted combination weights. Extensive experiments on multiple benchmarks demonstrate that LPC consistently improves upon three alignment algorithms (DPO, SimPO, and IPO) using three base models (Mistral-7B, Llama3-8B, and Llama3-8B-Instruct). Furthermore, deeper analysis reveals that the learned latent codes effectively capture the differences in the distribution of human preferences and significantly enhance the robustness of alignment against noise in data. By providing a unified representation for the multifarious preference factors, LPC paves the way towards developing more robust and versatile alignment techniques for the responsible deployment of powerful LLMs.

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes

TL;DR

Latent Preference Coding (LPC) introduces a discrete latent variable framework for aligning LLMs to human preferences by modeling multiple underlying factors with a codebook of latent codes. Using a prior and a posterior , LPC integrates with offline RLHF objectives (e.g., , , ) via a variational loss that combines likelihood over preferred pairs with a KL penalty. Empirical results across multiple base models and benchmarks show LPC improves downstream task performance and preference accuracy, and analyses reveal its latent codes capture diverse preference signals and offer robustness to noisy data. The approach provides a versatile, generalizable path toward more robust and nuanced alignment of powerful LLMs across tasks and populations.

Abstract

Large language models (LLMs) have achieved remarkable success, yet aligning their generations with human preferences remains a critical challenge. Existing approaches to preference modeling often rely on an explicit or implicit reward function, overlooking the intricate and multifaceted nature of human preferences that may encompass conflicting factors across diverse tasks and populations. To address this limitation, we introduce Latent Preference Coding (LPC), a novel framework that models the implicit factors as well as their combinations behind holistic preferences using discrete latent codes. LPC seamlessly integrates with various offline alignment algorithms, automatically inferring the underlying factors and their importance from data without relying on pre-defined reward functions and hand-crafted combination weights. Extensive experiments on multiple benchmarks demonstrate that LPC consistently improves upon three alignment algorithms (DPO, SimPO, and IPO) using three base models (Mistral-7B, Llama3-8B, and Llama3-8B-Instruct). Furthermore, deeper analysis reveals that the learned latent codes effectively capture the differences in the distribution of human preferences and significantly enhance the robustness of alignment against noise in data. By providing a unified representation for the multifarious preference factors, LPC paves the way towards developing more robust and versatile alignment techniques for the responsible deployment of powerful LLMs.
Paper Structure (34 sections, 22 equations, 2 figures, 7 tables)

This paper contains 34 sections, 22 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Latent Variable Models
  4. Learning from Human Feedback
  5. Multi-Objecitve Optimization
  6. Methodology
  7. Preliminaries: Reinforcement Learning from Human Feedback
  8. Learning Objective of Latent Preference Coding
  9. Modeling of Latent Preference Coding
  10. Discrete Latent Space.
  11. Posterior network.
  12. Prior network.
  13. Policy Model.
  14. Extension to Other Offline RLHF Objectives
  15. Experiments
  16. ...and 19 more sections

Figures (2)

  • Figure 1: Overview of Latent Preference Coding. The framework is comprised of a discrete codebook and three modules: a policy model $\pi_{\theta}(y|x,z)$ conditioned on a latent variable $z$, a prior network $p(z|x)$ that learns to infer $z$ from the prompt, and a posterior network $q(z|x,y_w\succ y_l)$ that guides the training of the prior network and latent code embeddings.
  • Figure 2: Top left: Preference accuracy (PA) of DPO w. LPC on Llama3-8B varying with the latent codebook size. Bottom Left: Flipping-label experiment on Llama3-8B. Models are evaluated on the original test set with unflipped labels. Right: Visualization of the latent variable $z$ produced by the prior network of Llama3-8B. The alignment method is DPO. For each data source in UltraFeedback, we randomly select 100 instances and visualize the T-SNE features of these instances.