Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining
Weiyi Wang, Junwei Deng, Yuzheng Hu, Shiyuan Zhang, Xirui Jiang, Runting Zhang, Han Zhao, Jiaqi W. Ma
TL;DR
Data attribution methods are highly sensitive to hyperparameters, and evaluating their performance for tuning is costly because it typically requires retraining on data subsets. The authors conduct the first large-scale empirical study across methods, datasets, and models, uncovering widespread hyperparameter sensitivity and nontrivial interactions between choices like the regularization $\lambda$ and projection dimension. They provide a theoretical analysis of the regularization term in influence-function-based methods and derive a retraining-free surrogate indicator, $\xi_{z',\lambda}$, to select $\lambda$ using full-data quantities. Across diverse settings and downstream tasks, the surrogate-indicator-based selection achieves strong, often near-optimal performance in $LDS$ and improves data-subset removal outcomes, offering a practical tool to mitigate a key deployment bottleneck in data attribution.
Abstract
Data attribution methods, which quantify the influence of individual training data points on a machine learning model, have gained increasing popularity in data-centric applications in modern AI. Despite a recent surge of new methods developed in this space, the impact of hyperparameter tuning in these methods remains under-explored. In this work, we present the first large-scale empirical study to understand the hyperparameter sensitivity of common data attribution methods. Our results show that most methods are indeed sensitive to certain key hyperparameters. However, unlike typical machine learning algorithms -- whose hyperparameters can be tuned using computationally-cheap validation metrics -- evaluating data attribution performance often requires retraining models on subsets of training data, making such metrics prohibitively costly for hyperparameter tuning. This poses a critical open challenge for the practical application of data attribution methods. To address this challenge, we advocate for better theoretical understandings of hyperparameter behavior to inform efficient tuning strategies. As a case study, we provide a theoretical analysis of the regularization term that is critical in many variants of influence function methods. Building on this analysis, we propose a lightweight procedure for selecting the regularization value without model retraining, and validate its effectiveness across a range of standard data attribution benchmarks. Overall, our study identifies a fundamental yet overlooked challenge in the practical application of data attribution, and highlights the importance of careful discussion on hyperparameter selection in future method development.