Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data

Jipeng Zhang, Yaxuan Qin, Renjie Pi, Weizhong Zhang, Rui Pan, Tong Zhang

TL;DR

TAGCOS tackles the data-efficiency challenge in instruction tuning by using per-sample gradient features as task-agnostic data representations, then applying gradient-based clustering to handle dataset diversity and a greedy optimal matching pursuit (OMP) strategy to select a compact, informative coreset. Theoretical analysis shows that this gradient-clustering decomposition preserves submodularity properties, enabling efficient optimization and yielding smaller coresets that approximate full-gradient objectives. Empirically, TAGCOS achieves near full-dataset performance with only 5% of the data across multiple benchmarks and transfers effectively between models, offering practical scalability for large language models. Overall, the approach provides a principled, scalable pipeline for data-efficient instruction tuning with strong empirical and theoretical support.

Abstract

Instruction tuning has achieved unprecedented success in NLP, turning large language models into versatile chatbots. However, the increasing variety and volume of instruction datasets demand significant computational resources. To address this, it is essential to extract a small and highly informative subset (i.e., Coreset) that achieves comparable performance to the full dataset. Achieving this goal poses non-trivial challenges: 1) data selection requires accurate data representations that reflect the training samples' quality, 2) considering the diverse nature of instruction datasets, and 3) ensuring the efficiency of the coreset selection algorithm for large models. To address these challenges, we propose Task-Agnostic Gradient Clustered COreset Selection (TAGCOS). Specifically, we leverage sample gradients as the data representations, perform clustering to group similar data, and apply an efficient greedy algorithm for coreset selection. Experimental results show that our algorithm, selecting only 5% of the data, surpasses other unsupervised methods and achieves performance close to that of the full dataset.

TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Data

TL;DR

TAGCOS tackles the data-efficiency challenge in instruction tuning by using per-sample gradient features as task-agnostic data representations, then applying gradient-based clustering to handle dataset diversity and a greedy optimal matching pursuit (OMP) strategy to select a compact, informative coreset. Theoretical analysis shows that this gradient-clustering decomposition preserves submodularity properties, enabling efficient optimization and yielding smaller coresets that approximate full-gradient objectives. Empirically, TAGCOS achieves near full-dataset performance with only 5% of the data across multiple benchmarks and transfers effectively between models, offering practical scalability for large language models. Overall, the approach provides a principled, scalable pipeline for data-efficient instruction tuning with strong empirical and theoretical support.

Abstract

Instruction tuning has achieved unprecedented success in NLP, turning large language models into versatile chatbots. However, the increasing variety and volume of instruction datasets demand significant computational resources. To address this, it is essential to extract a small and highly informative subset (i.e., Coreset) that achieves comparable performance to the full dataset. Achieving this goal poses non-trivial challenges: 1) data selection requires accurate data representations that reflect the training samples' quality, 2) considering the diverse nature of instruction datasets, and 3) ensuring the efficiency of the coreset selection algorithm for large models. To address these challenges, we propose Task-Agnostic Gradient Clustered COreset Selection (TAGCOS). Specifically, we leverage sample gradients as the data representations, perform clustering to group similar data, and apply an efficient greedy algorithm for coreset selection. Experimental results show that our algorithm, selecting only 5% of the data, surpasses other unsupervised methods and achieves performance close to that of the full dataset.
Paper Structure (14 sections, 2 theorems, 9 equations, 1 figure, 6 tables, 2 algorithms)

This paper contains 14 sections, 2 theorems, 9 equations, 1 figure, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Gradient Features Computation
  5. Gradient-based Clustering
  6. Coreset Selection via Optimal Matching Pursuit
  7. Theoretical Analysis
  8. Experiment
  9. Setup
  10. Experimental Results
  11. Ablation Study and Analysis
  12. Conclusion
  13. Limitation
  14. Training Dataset Details

Key Result

Lemma 1

If the coreset size $|D_{sub}| \leq c$ and $\max_{z \in D} \| \nabla_{\theta} \ell( z;\theta)\|_2 \leq G$, then $F_{\lambda}(D_{sub}; D)$ is $\gamma_D$-weakly submodular with $\gamma_D = \frac{\lambda}{\lambda + MG^2}$.

Figures (1)

  • Figure 1: Illustration of the proposed TAGCOS pipeline. Our framework consists of three stages: 1) Gradient feature computation, which efficiently derive sample-wise gradients to use as data representation; 2) Gradient-based Clustering, which groups data with high similarity into the same groups; 3) Coreset selection via OMP, which efficiently selects the coresets from each cluster separately in a greedy manner.

Theorems & Definitions (2)

  • Lemma 1
  • Theorem 1