Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sequence-level Large Language Model Training with Contrastive Preference Optimization

Zhili Feng, Dhananjay Ram, Cole Hawkins, Aditya Rawal, Jinman Zhao, Sheng Zha

TL;DR

This work tackles exposure bias in large language models by introducing Contrastive Preference Optimization (CPO), a sequence-level training objective that injects longer-horizon information without requiring human-labeled preferences. CPO contrasts a ground-truth prefix-continuation against multiple synthetic negatives drawn from various sampling schemes, optionally using ranking signals, and can be applied during supervision fine-tuning or late pretraining. Across Dolly instruction-following and Wikidump open-ended generation tasks, CPO improves win-rate against ground-truth continuations and outperforms baselines like MLE, DPO, and PSS, with further gains from weight-space ensembles. The approach offers a practical, computation-efficient path to align generation quality with sequence-level objectives, though synthetic negatives remain noisy unless generated autoregressively, pointing to future refinements in negative data generation.

Abstract

The next token prediction loss is the dominant self-supervised training objective for large language models and has achieved promising results in a variety of downstream tasks. However, upon closer investigation of this objective, we find that it lacks an understanding of sequence-level signals, leading to a mismatch between training and inference processes. To bridge this gap, we introduce a contrastive preference optimization (CPO) procedure that can inject sequence-level information into the language model at any training stage without expensive human labeled data. Our experiments show that the proposed objective surpasses the next token prediction in terms of win rate in the instruction-following and text generation tasks.

Sequence-level Large Language Model Training with Contrastive Preference Optimization

TL;DR

This work tackles exposure bias in large language models by introducing Contrastive Preference Optimization (CPO), a sequence-level training objective that injects longer-horizon information without requiring human-labeled preferences. CPO contrasts a ground-truth prefix-continuation against multiple synthetic negatives drawn from various sampling schemes, optionally using ranking signals, and can be applied during supervision fine-tuning or late pretraining. Across Dolly instruction-following and Wikidump open-ended generation tasks, CPO improves win-rate against ground-truth continuations and outperforms baselines like MLE, DPO, and PSS, with further gains from weight-space ensembles. The approach offers a practical, computation-efficient path to align generation quality with sequence-level objectives, though synthetic negatives remain noisy unless generated autoregressively, pointing to future refinements in negative data generation.

Abstract

The next token prediction loss is the dominant self-supervised training objective for large language models and has achieved promising results in a variety of downstream tasks. However, upon closer investigation of this objective, we find that it lacks an understanding of sequence-level signals, leading to a mismatch between training and inference processes. To bridge this gap, we introduce a contrastive preference optimization (CPO) procedure that can inject sequence-level information into the language model at any training stage without expensive human labeled data. Our experiments show that the proposed objective surpasses the next token prediction in terms of win rate in the instruction-following and text generation tasks.

Paper Structure

This paper contains 23 sections, 11 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Proposed approach
  4. Experimental Setup
  5. Task and model.
  6. Baselines.
  7. Training details.
  8. Evaluation.
  9. Weight-space ensemble.
  10. Experimental Analysis
  11. Instruction-Following Task
  12. Comparison with DPO and PSS.
  13. Sampling Strategy.
  14. Effect of different negative samples.
  15. Open-ended Text Generation Task
  16. ...and 8 more sections

Figures (2)

  • Figure 1: The effect of different generation configuration during inference and different negative sampling methods during training. Unless otherwise specified, greedy decoding is used. Win rate is evaluated by GPT-3.5 against the ground truth continuations.
  • Figure 2: MAUVE score of MLE and CPO on Wiki data.