Generalized Group Data Attribution

TL;DR

The Generalized Group Data Attribution framework is introduced, which computationally simplifies DA by attributing to groups of training points instead of individual ones, enabling practical applications in large-scale machine learning scenarios that were previously infeasible.

Abstract

Data Attribution (DA) methods quantify the influence of individual training data points on model outputs and have broad applications such as explainability, data selection, and noisy label identification. However, existing DA methods are often computationally intensive, limiting their applicability to large-scale machine learning models. To address this challenge, we introduce the Generalized Group Data Attribution (GGDA) framework, which computationally simplifies DA by attributing to groups of training points instead of individual ones. GGDA is a general framework that subsumes existing attribution methods and can be applied to new DA techniques as they emerge. It allows users to optimize the trade-off between efficiency and fidelity based on their needs. Our empirical results demonstrate that GGDA applied to popular DA methods such as Influence Functions, TracIn, and TRAK results in upto 10x-50x speedups over standard DA methods while gracefully trading off attribution fidelity. For downstream applications such as dataset pruning and noisy label identification, we demonstrate that GGDA significantly improves computational efficiency and maintains effectiveness, enabling practical applications in large-scale machine learning scenarios that were previously infeasible.

Figures (8)

• Figure 1: Ablation over grouping methods. Test accuracy (%) retraining score on MNIST after removing the top 20% most important training points (lower is better). Grad-K-Means grouping is superior, achieving comparable performance to individual attributions at orders of magnitudes faster runtime. Error bars represent standard error computed on 10 differently seeded model retrainings.
• Figure 2: GGDA attribution fidelity across removal percentages. Test accuracy (%) retraining score using Grad-K-Means grouping on CIFAR-10 after removing 1%, 5%, 10%, and 20% of the most important points (lower is better). GGDA methods save orders of magnitude of runtime while gracefully trading off attribution fidelity. See Appendix \ref{['subapp:retrain']} for similar findings across all datasets and models.
• Figure 3: Ablation over grouping methods. Top: LR. Middle: ANN-S. Bottom: ANN-M. Test accuracy (%) retraining score on HELOC after removing the top 20% most important training points (lower is better).
• Figure 4: Ablation over grouping methods. Test accuracy (%) retraining score on CIFAR-10 after removing the top 20% most important training points (lower is better).
• Figure 5: Ablation over grouping methods. Test accuracy (%) retraining score on QNLI after removing the top 20% most important training points (lower is better).

Theorems & Definitions (4)

• Definition 1
• Definition 2
• Definition 3
• Definition 4