Rethinking Data Selection for Supervised Fine-Tuning
Ming Shen
TL;DR
This paper challenges conventional data-selection wisdom for supervised fine-tuning of LLMs by arguing that SFT largely learns style rather than knowledge. It proposes a simple, human-style heuristic: select demonstrations with long, detailed responses ($|y_i|$) to form a small, highly effective fine-tuning subset. Experiments across Alpaca, WizardLM, and Dolly show that long-response subsets often outperform the full dataset as well as quality- and diversity-based selections, with strong gains on open-ended instruction benchmarks and long-form tasks. The findings suggest that curating data to reflect human-like interaction correlates with better instruction-following, and point to future work in broader, human-centered dataset curation and evaluation practices.
Abstract
Although supervised finetuning (SFT) has emerged as an essential technique to align large language models with humans, it is considered superficial, with style learning being its nature. At the same time, recent works indicate the importance of data selection for SFT, showing that finetuning with high-quality and diverse subsets of the original dataset leads to superior downstream performance. In this work, we rethink the intuition behind data selection for SFT. Considering SFT is superficial, we propose that essential demonstrations for SFT should focus on reflecting human-like interactions instead of data quality or diversity. However, it is not straightforward to directly assess to what extent a demonstration reflects human styles. Towards an initial attempt in this direction, we find selecting instances with long responses is surprisingly more effective for SFT than utilizing full datasets or instances selected based on quality and diversity. We hypothesize that such a simple heuristic implicitly mimics a crucial aspect of human-style conversation: detailed responses are usually more helpful.