Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficiently Estimating Data Efficiency for Language Model Fine-tuning

Gyung Hyun Je, Colin Raffel

TL;DR

This work tackles the problem of predicting how many fine-tuning examples a language model needs to achieve a target performance, a quantity highly variable across tasks. It defines data efficiency as the area under the task-specific performance curve over a maximum budget and introduces CoS-Low, a predictor based on the median cosine similarity of per-sample gradients computed on low-confidence examples, using only a small labeled set. By mapping a predicted AUC to a data-efficiency curve via a simple power function, the method can estimate the required data budget with low error (MAE ~0.086) and substantial cost savings across 30 real-world tasks. The approach generalizes across model families, extends to out-of-distribution tasks, and provides a practical tool to optimize annotation and fine-tuning effort in resource-constrained settings.

Abstract

While large language models (LLMs) demonstrate reasonable zero-shot capability across many downstream tasks, fine-tuning is a common practice to improve their performance. However, a task's data efficiency--i.e., the number of fine-tuning examples needed to achieve a desired level of performance--is often unknown, resulting in costly cycles of incremental annotation and retraining. Indeed, we demonstrate across a curated set of 30 specialized tasks that performant LLMs may struggle zero-shot but can attain stronger performance after fine-tuning. This motivates the need for methods to predict a task's data efficiency without requiring incremental annotation. After introducing a concrete metric that quantifies a task's data efficiency, we propose using the gradient cosine similarity of low-confidence examples to predict data efficiency based on a small number of labeled samples. We validate our approach on a diverse set of tasks with varying data efficiencies, attaining 8.6% error in overall data efficiency prediction and typically eliminating hundreds of unnecessary annotations on each task. Our experiment results and implementation code are available on GitHub.

Efficiently Estimating Data Efficiency for Language Model Fine-tuning

TL;DR

This work tackles the problem of predicting how many fine-tuning examples a language model needs to achieve a target performance, a quantity highly variable across tasks. It defines data efficiency as the area under the task-specific performance curve over a maximum budget and introduces CoS-Low, a predictor based on the median cosine similarity of per-sample gradients computed on low-confidence examples, using only a small labeled set. By mapping a predicted AUC to a data-efficiency curve via a simple power function, the method can estimate the required data budget with low error (MAE ~0.086) and substantial cost savings across 30 real-world tasks. The approach generalizes across model families, extends to out-of-distribution tasks, and provides a practical tool to optimize annotation and fine-tuning effort in resource-constrained settings.

Abstract

While large language models (LLMs) demonstrate reasonable zero-shot capability across many downstream tasks, fine-tuning is a common practice to improve their performance. However, a task's data efficiency--i.e., the number of fine-tuning examples needed to achieve a desired level of performance--is often unknown, resulting in costly cycles of incremental annotation and retraining. Indeed, we demonstrate across a curated set of 30 specialized tasks that performant LLMs may struggle zero-shot but can attain stronger performance after fine-tuning. This motivates the need for methods to predict a task's data efficiency without requiring incremental annotation. After introducing a concrete metric that quantifies a task's data efficiency, we propose using the gradient cosine similarity of low-confidence examples to predict data efficiency based on a small number of labeled samples. We validate our approach on a diverse set of tasks with varying data efficiencies, attaining 8.6% error in overall data efficiency prediction and typically eliminating hundreds of unnecessary annotations on each task. Our experiment results and implementation code are available on GitHub.
Paper Structure (35 sections, 9 equations, 12 figures, 10 tables, 1 algorithm)

This paper contains 35 sections, 9 equations, 12 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Establishing the Variability of Data Efficiency
  3. Measuring Data Efficiency
  4. Defining Data Efficiency
  5. Predicting Data Efficiency
  6. Mapping Task Difficulty to Data Efficiency
  7. Experimental Setup
  8. Experimental Results
  9. AUC Prediction Accuracy
  10. Fine-tuning Data Size Prediction Accuracy
  11. Ablation Studies and Further Analysis
  12. Does CoS-Low display strong correlation with data efficiency across models families of varying sizes?
  13. Can CoS-Low reliably predict data efficiencies of out-of-distribution tasks?
  14. How does CoS-Low evolve as the model is further trained on the target task?
  15. When is CoS-Low's core assumption not met?
  16. ...and 20 more sections

Figures (12)

  • Figure 1: Overview of our approach to predict task-specific data efficiency from a few labeled data points. We formally define a task's "data efficiency" as the area under its data efficiency curve (left). Then, we show that the aggregate gradient cosine similarity among low-confidence examples (CoS-Low) is a strong predictor of data efficiency. We propose a method leveraging CoS-Low to predict the task's data efficiency and its data efficiency curve, providing a concrete estimation of the fine-tuning data size needed to reach a target performance.
  • Figure 2: Comparing data budget (from 0 to 5000 examples on log-scale, x-axis) and task performance (from zero-shot to human-level performance, y-axis) across the 30 downstream tasks. The plots are sorted by speed of convergence to the maximum performance level as the fine-tuning data size increases.
  • Figure 3: CoS-Low (right) shows the strongest relationship with task data efficiency among other task difficulty metrics. Each metric is compared with the ground-truth task data efficiency (y-axis) using Spearman's rank correlation.
  • Figure 4: CoS-Low's correlation with task data efficiency and the AUC prediction error for the held-in and OOD tasks.
  • Figure 5: Relationship between zero-shot accuracy and task data efficiency. While higher zero-shot accuracy of tasks close to performance saturation may indicate lower task data accuracy, the relationship is not consistent (Spearman rank correlation of -0.3139).
  • ...and 7 more figures