FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
Rohan Deb, Kiran Thekumparampil, Kousha Kalantari, Gaurush Hiranandani, Shoham Sabach, Branislav Kveton
TL;DR
FisherSFT tackles the data efficiency challenge in supervised fine-tuning of large language models by selecting the most informative training sentences. It reframes sentence selection as a greedy optimal design problem that maximizes a lower bound on the log-determinant of the Hessian of the log-likelihood, enabling efficient estimation using a last-layer linearization with multinomial logistic regression. The paper provides a theoretical error bound of order $O(1/\sqrt{n})$ under mild assumptions and introduces a fast Greedy design variant with cached gains for scalability. Empirically, FisherSFT outperforms baselines on synthetic tasks, word embeddings, and GPT-2 Shakespeare data, and yields higher-quality text according to LLM-based evaluation, highlighting its practical impact for data-efficient language model adaptation.
Abstract
Supervised fine-tuning (SFT) is a standard approach to adapting large language models (LLMs) to new domains. In this work, we improve the statistical efficiency of SFT by selecting an informative subset of training examples. Specifically, for a fixed budget of training examples, which determines the computational cost of fine-tuning, we determine the most informative ones. The key idea in our method is to select examples that maximize information gain, measured by the Hessian of the log-likelihood of the LLM. We approximate it efficiently by linearizing the LLM at the last layer using multinomial logistic regression models. Our approach is computationally efficient, analyzable, and performs well empirically. We demonstrate this on several problems, and back our claims with both quantitative results and an LLM evaluation.