Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Study of SFT-DPO Interaction and Parameterization in Small Language Models

Yuming Feng, Christy Yang

Abstract

Direct Preference Optimization (DPO) is widely used after supervised fine-tuning (SFT) to align language models, yet empirical behavior under small backbones and modest data is under-specified. We systematically compare SFT-only, DPO-only, and staged SFT-to-DPO training alongside full fine-tuning (FFT) versus LoRA on a GPT-2-scale decoder, evaluating paraphrase detection and Shakespearean sonnet continuation. DPO yields small, task-dependent gains over strong SFT and can match competitive SFT accuracy without a warm start when the preference construction closely parallels the supervised objective. In contrast, parameterization dominates: FFT consistently outperforms LoRA at matched training depth, and LoRA does not reduce wall-clock time on our hardware. These findings indicate that, in this small-scale regime, supervised full-parameter adaptation remains the primary performance lever, while preference optimization and low-rank adaptation provide limited marginal returns.

An Empirical Study of SFT-DPO Interaction and Parameterization in Small Language Models

Abstract

Direct Preference Optimization (DPO) is widely used after supervised fine-tuning (SFT) to align language models, yet empirical behavior under small backbones and modest data is under-specified. We systematically compare SFT-only, DPO-only, and staged SFT-to-DPO training alongside full fine-tuning (FFT) versus LoRA on a GPT-2-scale decoder, evaluating paraphrase detection and Shakespearean sonnet continuation. DPO yields small, task-dependent gains over strong SFT and can match competitive SFT accuracy without a warm start when the preference construction closely parallels the supervised objective. In contrast, parameterization dominates: FFT consistently outperforms LoRA at matched training depth, and LoRA does not reduce wall-clock time on our hardware. These findings indicate that, in this small-scale regime, supervised full-parameter adaptation remains the primary performance lever, while preference optimization and low-rank adaptation provide limited marginal returns.
Paper Structure (42 sections, 4 equations, 1 figure, 13 tables)

This paper contains 42 sections, 4 equations, 1 figure, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language Model Fine-Tuning
  4. Preference-Based Optimization
  5. Parameter-Efficient Fine-Tuning
  6. Our Work
  7. Task Formulation
  8. Paraphrase Detection
  9. Sonnet Generation
  10. Approach
  11. Base Model
  12. Task Adaptation
  13. Supervised Fine-Tuning
  14. Direct Preference Optimization
  15. Parameterization Strategies
  16. ...and 27 more sections

Figures (1)

  • Figure 1: Training curves for FFT and LoRA ($r=8$) on paraphrase detection. Plot train and dev accuracy/F1 across epochs.