Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Pair Construction for Contrastive Post-training

Canwen Xu, Corby Rosset, Ethan C. Chau, Luciano Del Corro, Shweti Mahajan, Julian McAuley, Jennifer Neville, Ahmed Hassan Awadallah, Nikhil Rao

TL;DR

This work tackles the problem of aligning large language models without heavy reliance on human-provided preferences by automatically constructing contrastive training data from outputs of models with different capabilities. It analyzes and contrasts RLHF, SLiC, and DPO, highlighting that offline contrastive methods, especially DPO, offer stable and scalable improvements over SFT, with a data curriculum further boosting performance. The authors demonstrate that DPO trained on GPT-4 versus InstructGPT pairs yields strong gains on standard benchmarks and that incorporating this approach into Orca-13B pushes performance beyond ChatGPT. The results suggest a practical pathway to scalable alignment through automatic pair construction and curriculum learning, with meaningful implications for model distillation and deployment at scale.

Abstract

Alignment serves as an important step to steer large language models (LLMs) towards human preferences. In this paper, we propose an automatic way to construct contrastive data for LLM, using preference pairs from multiple models of varying strengths (e.g., InstructGPT, ChatGPT and GPT-4). We compare the contrastive techniques of SLiC and DPO to SFT baselines and find that DPO provides a step-function improvement even after continuing SFT saturates. We also explore a data curriculum learning scheme for contrastive post-training, which starts by learning from "easier" pairs and transitioning to "harder" ones, which further improves alignment. Finally, we scale up our experiments to train with more data and larger models like Orca. Remarkably, our automatic contrastive post-training further improves the performance of Orca, already a state-of-the-art instruction learning model tuned with GPT-4 outputs, to outperform ChatGPT.

Automatic Pair Construction for Contrastive Post-training

TL;DR

This work tackles the problem of aligning large language models without heavy reliance on human-provided preferences by automatically constructing contrastive training data from outputs of models with different capabilities. It analyzes and contrasts RLHF, SLiC, and DPO, highlighting that offline contrastive methods, especially DPO, offer stable and scalable improvements over SFT, with a data curriculum further boosting performance. The authors demonstrate that DPO trained on GPT-4 versus InstructGPT pairs yields strong gains on standard benchmarks and that incorporating this approach into Orca-13B pushes performance beyond ChatGPT. The results suggest a practical pathway to scalable alignment through automatic pair construction and curriculum learning, with meaningful implications for model distillation and deployment at scale.

Abstract

Alignment serves as an important step to steer large language models (LLMs) towards human preferences. In this paper, we propose an automatic way to construct contrastive data for LLM, using preference pairs from multiple models of varying strengths (e.g., InstructGPT, ChatGPT and GPT-4). We compare the contrastive techniques of SLiC and DPO to SFT baselines and find that DPO provides a step-function improvement even after continuing SFT saturates. We also explore a data curriculum learning scheme for contrastive post-training, which starts by learning from "easier" pairs and transitioning to "harder" ones, which further improves alignment. Finally, we scale up our experiments to train with more data and larger models like Orca. Remarkably, our automatic contrastive post-training further improves the performance of Orca, already a state-of-the-art instruction learning model tuned with GPT-4 outputs, to outperform ChatGPT.
Paper Structure (23 sections, 3 equations, 1 figure, 7 tables)

This paper contains 23 sections, 3 equations, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Reinforcement Learning from Human Feedback (RLHF)
  5. Sequence Likelihood Calibration (SLiC)
  6. Direct Preference Optimization (DPO)
  7. Contrastive Post-training over Pairwise Data Curriculum
  8. Contrastive Post-training
  9. Data Curriculum
  10. Experiments
  11. Experimental Settings
  12. Training Datasets
  13. Evaluation Datasets
  14. Base Models
  15. Training Details
  16. ...and 8 more sections

Figures (1)

  • Figure 1: Difference betwen SFT, RLHF, and contrastive post-training. For SFT, the model optimizes the negative log-likelihood for the next token. RLHF samples an output from the LLM and use a reward model to provide feedback for PPO to update the LLM. For contrastive post-training, a contrastive loss is used to steer the model towards preferred outputs.