Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling

Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, Kaiqi Huang

TL;DR

Sequential Preference Optimization (SPO) tackles multi-dimensional human preference alignment for LLMs by performing sequential, constrained fine-tuning that preserves prior dimensions while optimizing new ones, avoiding explicit multi-reward models. The authors derive a closed-form optimal policy for two dimensions and generalize to arbitrary dimensions with a tractable loss, plus gradient analysis showing how prior alignment is regularized. Empirical results on PKU-SafeRLHF-30k and Helpsteer2 with Llama 2 bases demonstrate SPO’s ability to outperform baselines, resist overfitting, and scale to more dimensions. This approach offers a practical, efficient path to robust multi-dimensional alignment in real-world systems using implicit reward signaling and LoRA-enabled fine-tuning.

Abstract

Human preference alignment is critical in building powerful and reliable large language models (LLMs). However, current methods either ignore the multi-dimensionality of human preferences (e.g. helpfulness and harmlessness) or struggle with the complexity of managing multiple reward models. To address these issues, we propose Sequential Preference Optimization (SPO), a method that sequentially fine-tunes LLMs to align with multiple dimensions of human preferences. SPO avoids explicit reward modeling, directly optimizing the models to align with nuanced human preferences. We theoretically derive closed-form optimal SPO policy and loss function. Gradient analysis is conducted to show how SPO manages to fine-tune the LLMs while maintaining alignment on previously optimized dimensions. Empirical results on LLMs of different size and multiple evaluation datasets demonstrate that SPO successfully aligns LLMs across multiple dimensions of human preferences and significantly outperforms the baselines.

SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling

TL;DR

Sequential Preference Optimization (SPO) tackles multi-dimensional human preference alignment for LLMs by performing sequential, constrained fine-tuning that preserves prior dimensions while optimizing new ones, avoiding explicit multi-reward models. The authors derive a closed-form optimal policy for two dimensions and generalize to arbitrary dimensions with a tractable loss, plus gradient analysis showing how prior alignment is regularized. Empirical results on PKU-SafeRLHF-30k and Helpsteer2 with Llama 2 bases demonstrate SPO’s ability to outperform baselines, resist overfitting, and scale to more dimensions. This approach offers a practical, efficient path to robust multi-dimensional alignment in real-world systems using implicit reward signaling and LoRA-enabled fine-tuning.

Abstract

Human preference alignment is critical in building powerful and reliable large language models (LLMs). However, current methods either ignore the multi-dimensionality of human preferences (e.g. helpfulness and harmlessness) or struggle with the complexity of managing multiple reward models. To address these issues, we propose Sequential Preference Optimization (SPO), a method that sequentially fine-tunes LLMs to align with multiple dimensions of human preferences. SPO avoids explicit reward modeling, directly optimizing the models to align with nuanced human preferences. We theoretically derive closed-form optimal SPO policy and loss function. Gradient analysis is conducted to show how SPO manages to fine-tune the LLMs while maintaining alignment on previously optimized dimensions. Empirical results on LLMs of different size and multiple evaluation datasets demonstrate that SPO successfully aligns LLMs across multiple dimensions of human preferences and significantly outperforms the baselines.
Paper Structure (12 sections, 9 equations, 2 figures, 7 tables)

This paper contains 12 sections, 9 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Related Work
  4. Methodology
  5. Two-Dimensional Sequential Alignment
  6. Multi-Dimensional Sequential Alignment
  7. Experiment
  8. Experiment Setting
  9. Results with the Modified Training Dataset
  10. Results with the Original Training Dateset
  11. Experiments with More Preference Dimensions
  12. Conclusion and Limitation

Figures (2)

  • Figure 1: The SFT model is sequentially fine-tuned on multi-dimensional preferences with SPO, which aligns LLMs on the current dimension and preserves alignment on previous dimensions. First-round fine-tuning is achieved by DPO as there is no constraint on previous alignments.
  • Figure 2: (a) Scores on harmlessness and helpfulness evaluation datasets by GPT-4 evaluator. (b) Aggregated utility of two dimensions, which is the product of harmlessness and helpfulness scores. (c), (d) give Helpfulness and Harmlessness rewards during the second-round fine-tuning process of SPO and S-DPO. SPO better preserves alignment on the first dimension of Helpfulness while learning to align with Harmlessness.