Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty-Aware Gradient Signal-to-Noise Data Selection for Instruction Tuning

Zhihang Yuan, Chengyu Yue, Long Huang, Litu Ou, Lei Shi

TL;DR

Instruction tuning datasets are large, noisy, and redundant, making full-data fine-tuning costly. GradFiltering introduces an uncertainty-aware, gradient-based data selection framework that uses a LoRA ensemble on a frozen backbone and a small GPT-2 proxy to compute per-example gradients, summarized as the Gradient Signal-to-Noise Ratio (G-SNR). By combining early-to-late gradient drops with ensemble disagreement, G-SNR ranks data points in an objective-agnostic way and yields high-quality subsets that often match or exceed full-data performance while converging faster. Empirical results on Alpaca and Alpaca-GPT4 with LLaMA-2 backbones show robust gains over random and strong baselines, with human judgments corroborating the improvements, highlighting the practical impact for scalable instruction tuning.

Abstract

Instruction tuning is a standard paradigm for adapting large language models (LLMs), but modern instruction datasets are large, noisy, and redundant, making full-data fine-tuning costly and often unnecessary. Existing data selection methods either build expensive gradient datastores or assign static scores from a weak proxy, largely ignoring evolving uncertainty, and thus missing a key source of LLM interpretability. We propose GRADFILTERING, an objective-agnostic, uncertainty-aware data selection framework that utilizes a small GPT-2 proxy with a LoRA ensemble and aggregates per-example gradients into a Gradient Signal-to-Noise Ratio (G-SNR) utility. Our method matches or surpasses random subsets and strong baselines in most LLM-as-a-judge evaluations as well as in human assessment. Moreover, GRADFILTERING-selected subsets converge faster than competitive filters under the same compute budget, reflecting the benefit of uncertainty-aware scoring.

Uncertainty-Aware Gradient Signal-to-Noise Data Selection for Instruction Tuning

TL;DR

Instruction tuning datasets are large, noisy, and redundant, making full-data fine-tuning costly. GradFiltering introduces an uncertainty-aware, gradient-based data selection framework that uses a LoRA ensemble on a frozen backbone and a small GPT-2 proxy to compute per-example gradients, summarized as the Gradient Signal-to-Noise Ratio (G-SNR). By combining early-to-late gradient drops with ensemble disagreement, G-SNR ranks data points in an objective-agnostic way and yields high-quality subsets that often match or exceed full-data performance while converging faster. Empirical results on Alpaca and Alpaca-GPT4 with LLaMA-2 backbones show robust gains over random and strong baselines, with human judgments corroborating the improvements, highlighting the practical impact for scalable instruction tuning.

Abstract

Instruction tuning is a standard paradigm for adapting large language models (LLMs), but modern instruction datasets are large, noisy, and redundant, making full-data fine-tuning costly and often unnecessary. Existing data selection methods either build expensive gradient datastores or assign static scores from a weak proxy, largely ignoring evolving uncertainty, and thus missing a key source of LLM interpretability. We propose GRADFILTERING, an objective-agnostic, uncertainty-aware data selection framework that utilizes a small GPT-2 proxy with a LoRA ensemble and aggregates per-example gradients into a Gradient Signal-to-Noise Ratio (G-SNR) utility. Our method matches or surpasses random subsets and strong baselines in most LLM-as-a-judge evaluations as well as in human assessment. Moreover, GRADFILTERING-selected subsets converge faster than competitive filters under the same compute budget, reflecting the benefit of uncertainty-aware scoring.
Paper Structure (40 sections, 5 equations, 3 figures, 3 tables)

This paper contains 40 sections, 5 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Our approach.
  3. Related Work
  4. Data Valuation and Selection
  5. Ensembles and Uncertainty Modeling
  6. Positioning our approach.
  7. LoRA-Ensemble Approximation for Gradient-Based Data Valuation
  8. LoRA-Ensemble Approximation
  9. Gradient-based view of LoRA ensembles.
  10. Empirical geometry of LoRA training dynamics.
  11. Practical Considerations
  12. GradFiltering : Data Selection with G-SNR
  13. Problem Setup
  14. Per-Example Gradient Statistics
  15. Gradient magnitude.
  16. ...and 25 more sections

Figures (3)

  • Figure 1: Trajectories of LoRA ensemble members in a low-dimensional embedding of gradient profiles for (a) Alpaca and (b) Alpaca-GPT4. Each polyline corresponds to one LoRA member; markers along the line denote successive epochs.
  • Figure 2: Overview of our gradient-based data selection pipeline. Phase 1 trains a set of LoRA-ensemble members that share the same frozen backbone $\theta_0$ on the full dataset $\mathcal{D}$, producing multiple adapters $\{\Delta\theta_e^{(m)}\}_{m=1}^M$ after $e$ epochs. Phase 2 collects per-example gradients respect to $\Delta\theta$ during training process at different epochs, yielding gradient profiles $\{g_i^{(m,e)}\}_{m=1}^M$. Phase 3 calculates an uncertainty-aware utility score based on these profiles by combining (i) an information-gain term $\Delta G_i = G_i^{(s)} - G_i^{(t)}$ that measures gradient drop from early to later training, and (ii) a disagreement term $V_i^{(t)}=\mathop{\mathrm{Var}}\limits_{m}\!(\|g_i^{(m,t)}\|_2)$ that captures ensemble variability, via an SNR-style form $u_i \propto \frac{\Delta G_i}{G_i^{(s)}+\epsilon}\cdot\frac{1}{V_i^{(t)}+\epsilon}$. Phase 4 selects the top-ranked subset $\mathcal{D}'\subset\mathcal{D}$ based on the utility score $u_i$ for training the targeted model.
  • Figure 3: Training loss for LLaMA-2-13B on Alpaca with 10% of the data. Left: full-parameter fine-tuning; right: LoRA-based fine-tuning. In both settings, we compare convergence speed and final training loss between our selection (ours-10%) and Superfiltering-10%.