Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Influential Language Data Selection via Gradient Trajectory Pursuit

Zhiwei Deng, Tao Li, Yang Li

TL;DR

This paper proposes Gradient Trajectory Pursuit (GTP), an algorithm that performs pursuit of gradient trajectories via jointly selecting data points under an L0-norm regularized objective and highlights higher efficiency with compressive sampling processes, which can be further sped up using a distributed framework.

Abstract

Curating a desirable dataset for training has been the core of building highly capable large language models (Touvron et al., 2023; Achiam et al., 2023; Team et al.,2024). Gradient influence scores (Pruthi et al., 2020; Xia et al., 2024) are shown to be correlated with model performance and are commonly used as the criterion for data selection. However, existing methods are built upon either individual sample rankings or inefficient matching process, leading to suboptimal performance or scaling up issues.In this paper, we propose Gradient Trajectory Pursuit (GTP), an algorithm that performs pursuit of gradient trajectories via jointly selecting data points under an L0-norm regularized objective. The proposed algorithm highlights: (1) joint selection instead of independent top-k selection, which automatically de-duplicates samples; (2) higher efficiency with compressive sampling processes, which can be further sped up using a distributed framework. In the experiments, we demonstrate the algorithm in both in-domain and target-domain selection benchmarks and show that it outperforms top-k selection and competitive algorithms consistently, for example, our algorithm chooses as low as 0.5% data to achieve full performance on the targeted instruction tuning tasks

Influential Language Data Selection via Gradient Trajectory Pursuit

TL;DR

This paper proposes Gradient Trajectory Pursuit (GTP), an algorithm that performs pursuit of gradient trajectories via jointly selecting data points under an L0-norm regularized objective and highlights higher efficiency with compressive sampling processes, which can be further sped up using a distributed framework.

Abstract

Curating a desirable dataset for training has been the core of building highly capable large language models (Touvron et al., 2023; Achiam et al., 2023; Team et al.,2024). Gradient influence scores (Pruthi et al., 2020; Xia et al., 2024) are shown to be correlated with model performance and are commonly used as the criterion for data selection. However, existing methods are built upon either individual sample rankings or inefficient matching process, leading to suboptimal performance or scaling up issues.In this paper, we propose Gradient Trajectory Pursuit (GTP), an algorithm that performs pursuit of gradient trajectories via jointly selecting data points under an L0-norm regularized objective. The proposed algorithm highlights: (1) joint selection instead of independent top-k selection, which automatically de-duplicates samples; (2) higher efficiency with compressive sampling processes, which can be further sped up using a distributed framework. In the experiments, we demonstrate the algorithm in both in-domain and target-domain selection benchmarks and show that it outperforms top-k selection and competitive algorithms consistently, for example, our algorithm chooses as low as 0.5% data to achieve full performance on the targeted instruction tuning tasks

Paper Structure

This paper contains 10 sections, 3 equations, 5 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Our method
  4. Problem statement
  5. Gradient Trajectory Pursuit
  6. Computation time analysis
  7. Experiments
  8. LLM Agent: ALFWorld
  9. Targeted instruction tuning
  10. Conclusions

Figures (5)

  • Figure 1: Our method selects a subset of training data through pursuit process of gradient trajectories on a target subspace during warmup training. The algorithm automatically de-duplicates samples instead of simply selecting the top-$k$ data. Text examples in the figure are drawn from instruction tuning datasets.
  • Figure 2: Three examples with topmost scores from selection algorithms. Left: Top-$k$ tends to select redundant samples due to lack of de-duplication mechanisms. Right: GTP instead chooses diverse examples.
  • Figure 3: Residual norms across iterations.
  • Figure 4: Subset accuracy across iterations.
  • Figure :