Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Layer-Aware Influence for Online Data Valuation Estimation

Ziao Yang, Longbo Huang, Hongfu Liu

TL;DR

The paper tackles online data valuation by introducing Layer-Aware Influence (LAI), a Hessian-free estimator that uses only loss-to-output gradients and layer-embedded similarities to score training samples during optimization. LAI achieves high fidelity to Shapley values while dramatically reducing time and memory costs, enabling dynamic data curation within SGD/Adam pipelines. Empirical results across LLM pre-training/fine-tuning and image/text classification demonstrate improved generalization and reduced computation, with LAI often outperforming the more complex Ghost influence and other baselines. The work provides practical, scalable tools for data-centric learning and highlights the value of accounting for evolving sample influence during training.

Abstract

Data-centric learning emphasizes curating high-quality training samples to boost performance rather than designing new architectures. A central problem is to estimate the influence of training sample efficiently. Prior studies largely focus on static influence measured on a converged model, overlooking how data valuation dynamically changes during optimization. This omission neglects the dynamic nature of sample influence during optimization, especially in deep models. To address the computational burden of frequent influence estimation, we develop a layer-aware online estimator that requires only loss-to-output gradients. This design avoids parameter-level and full-network gradients while preserving ranking fidelity. Extensive experiments across LLM pretraining, fine-tuning, and image classification show our method improves accuracy with substantially lower time and memory cost, making dynamic data curation efficient and scalable in practice.

Layer-Aware Influence for Online Data Valuation Estimation

TL;DR

The paper tackles online data valuation by introducing Layer-Aware Influence (LAI), a Hessian-free estimator that uses only loss-to-output gradients and layer-embedded similarities to score training samples during optimization. LAI achieves high fidelity to Shapley values while dramatically reducing time and memory costs, enabling dynamic data curation within SGD/Adam pipelines. Empirical results across LLM pre-training/fine-tuning and image/text classification demonstrate improved generalization and reduced computation, with LAI often outperforming the more complex Ghost influence and other baselines. The work provides practical, scalable tools for data-centric learning and highlights the value of accounting for evolving sample influence during training.

Abstract

Data-centric learning emphasizes curating high-quality training samples to boost performance rather than designing new architectures. A central problem is to estimate the influence of training sample efficiently. Prior studies largely focus on static influence measured on a converged model, overlooking how data valuation dynamically changes during optimization. This omission neglects the dynamic nature of sample influence during optimization, especially in deep models. To address the computational burden of frequent influence estimation, we develop a layer-aware online estimator that requires only loss-to-output gradients. This design avoids parameter-level and full-network gradients while preserving ranking fidelity. Extensive experiments across LLM pretraining, fine-tuning, and image classification show our method improves accuracy with substantially lower time and memory cost, making dynamic data curation efficient and scalable in practice.
Paper Structure (40 sections, 37 equations, 2 figures, 4 tables)

This paper contains 40 sections, 37 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries on Influence Functions
  4. Hessian-free Influence Functions
  5. Ghost of Hessian-free Influence Functions
  6. Methods
  7. Layer-aware influence estimator
  8. Advantages of LAI
  9. Experiments on LLMs
  10. Fidelity Validation
  11. LAI for Pre-training LLM
  12. LAI for Fine-tuning LLM
  13. Experiments on Image and Text Classification
  14. Algorithmic Performance
  15. Exploration on LAI at sample level
  16. ...and 25 more sections

Figures (2)

  • Figure 1: Fidelity Validation of LAI and ghost influence. A-D show representative checkpoints at steps 2,500, 5,000, 7,500, and 10,000: each point plots a sample’s proxy score (LAI is blue, Ghost is orange) versus its Shapley value at that step, with dashed least-squares fits. E reports the per-step Pearson correlation across all 100 checkpoints. F summarizes the distribution of these 100 per-step correlations via violin plots with dashed mean lines.
  • Figure 2: Sample influence dynamics of LAI on CIFAR10-N-w. A illustrates the distribution of influence scores across different epochs; B visualizes sample involvement across epochs, with blue indicating inclusion in a batch and white indicating exclusion; C shows the dynamic nature by tracking the influence scores of three samples across epochs.