Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning

Xiaotian Lin, Yanlin Qi, Yizhang Zhu, Themis Palpanas, Chengliang Chai, Nan Tang, Yuyu Luo

TL;DR

LEAD addresses the overhead of iterative model-aware data selection for LLM instruction tuning by estimating per-sample utility entirely within the normal training loop. It introduces Instance-Level Dynamic Uncertainty ($IDU$), which fuses current loss, gradient-based loss-change prediction, and historical smoothing to produce stable, zero-cost sample utilities. A two-stage coarse-to-fine strategy employs offline clustering by instruction difficulty and task similarity, combined with online EXP3-based scheduling and $IDU$-driven fine-grained selection. Theoretical results derive the optimal smoothing coefficient under a budget constraint and a gradient-based approximation for loss changes, while experiments show LEAD improves average performance by 6.1%–10.8% using only 2.5% of data and reduces training time by 5–10× across multiple models and benchmarks. This approach provides practical efficiency gains for instruction tuning, enabling high-quality data selection at scale without additional inference overhead.

Abstract

Instruction tuning has emerged as a critical paradigm for improving the capabilities and alignment of large language models (LLMs). However, existing iterative model-aware data selection methods incur significant computational overhead, as they rely on repeatedly performing full-dataset model inference to estimate sample utility for subsequent training iterations, creating a fundamental efficiency bottleneck. In this paper, we propose LEAD, an efficient iterative data selection framework that accurately estimates sample utility entirely within the standard training loop, eliminating the need for costly additional model inference. At its core, LEAD introduces Instance-Level Dynamic Uncertainty (IDU), a theoretically grounded utility function combining instantaneous training loss, gradient-based approximation of loss changes, and exponential smoothing of historical loss signals. To further scale efficiently to large datasets, LEAD employs a two-stage, coarse-to-fine selection strategy, adaptively prioritizing informative clusters through a multi-armed bandit mechanism, followed by precise fine-grained selection of high-utility samples using IDU. Extensive experiments across four diverse benchmarks show that LEAD significantly outperforms state-of-the-art methods, improving average model performance by 6.1%-10.8% while using only 2.5% of the training data and reducing overall training time by 5-10x.

LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning

TL;DR

LEAD addresses the overhead of iterative model-aware data selection for LLM instruction tuning by estimating per-sample utility entirely within the normal training loop. It introduces Instance-Level Dynamic Uncertainty (), which fuses current loss, gradient-based loss-change prediction, and historical smoothing to produce stable, zero-cost sample utilities. A two-stage coarse-to-fine strategy employs offline clustering by instruction difficulty and task similarity, combined with online EXP3-based scheduling and -driven fine-grained selection. Theoretical results derive the optimal smoothing coefficient under a budget constraint and a gradient-based approximation for loss changes, while experiments show LEAD improves average performance by 6.1%–10.8% using only 2.5% of data and reduces training time by 5–10× across multiple models and benchmarks. This approach provides practical efficiency gains for instruction tuning, enabling high-quality data selection at scale without additional inference overhead.

Abstract

Instruction tuning has emerged as a critical paradigm for improving the capabilities and alignment of large language models (LLMs). However, existing iterative model-aware data selection methods incur significant computational overhead, as they rely on repeatedly performing full-dataset model inference to estimate sample utility for subsequent training iterations, creating a fundamental efficiency bottleneck. In this paper, we propose LEAD, an efficient iterative data selection framework that accurately estimates sample utility entirely within the standard training loop, eliminating the need for costly additional model inference. At its core, LEAD introduces Instance-Level Dynamic Uncertainty (IDU), a theoretically grounded utility function combining instantaneous training loss, gradient-based approximation of loss changes, and exponential smoothing of historical loss signals. To further scale efficiently to large datasets, LEAD employs a two-stage, coarse-to-fine selection strategy, adaptively prioritizing informative clusters through a multi-armed bandit mechanism, followed by precise fine-grained selection of high-utility samples using IDU. Extensive experiments across four diverse benchmarks show that LEAD significantly outperforms state-of-the-art methods, improving average model performance by 6.1%-10.8% while using only 2.5% of the training data and reducing overall training time by 5-10x.
Paper Structure (29 sections, 7 theorems, 39 equations, 11 figures, 4 tables)

This paper contains 29 sections, 7 theorems, 39 equations, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Stage 1: Coarse-level Cluster Selection.
  3. Stage 2: Fine-Grained Sample Utility Estimation and Selection.
  4. Preliminary and Problem Formulation
  5. Instruction Tuning for LLMs
  6. Data Selection for Instruction Tuning
  7. Problem Formulation
  8. Instance-Level Dynamic Uncertainty Utility
  9. LEAD: LEArning to Iteratively Select Data
  10. LEAD Framework: An Overview
  11. LEAD Framework: Core Components
  12. Training Iteration Workflow of LEAD
  13. The Design Details of LEAD
  14. Instance-Level Dynamic Uncertainty Optimization under the Budget Constraint
  15. Adaptive Data Selection via MAB-Integrated Training Scheduler
  16. ...and 14 more sections

Key Result

theorem 1

For a given sample subset $S_i$, the utility change from parameter update $\theta_{t-1}$ to $\theta_t$ can be approximated as: where $\eta$ is the learning rate, $\delta_{t_k}$ and $\delta_{t-1}$ denote historical gradient norms, and $\phi$ is the angle between consecutive gradient directions, given by: $\cos\phi=\frac{\Delta\theta_{t_k}^{\top}\Delta\theta_{t-1}}{\|\Delta\theta_{t_k}\|\cdot\|\Del

Figures (11)

  • Figure 1: Comparison of Iterative Model-Aware Solutions.
  • Figure 1: Comparison of Performance across Different Benchmarks for Various Methods.
  • Figure 2: A High-level Overview of LEAD.
  • Figure 3: An Overview of the LEAD Framework.
  • Figure 4: Iterative Sample Selection Guided by IDU Scores.
  • ...and 6 more figures

Theorems & Definitions (8)

  • definition 1: Iterative Data Selection with Inference-Free Utility Estimation
  • theorem 1: Utility Change Approximation
  • theorem 2: Optimal Weight $\beta^*$
  • theorem 3: Approximation Error Bound
  • theorem 4: Optimal Smoothing Coefficient
  • lemma 1: Batch Utility Change Decomposition
  • lemma 2: Expected Sample Size Under MAB mechanism
  • theorem 5: IU Change Approximation