Training Data Influence Analysis and Estimation: A Survey
Zayd Hammoudeh, Daniel Lowd
TL;DR
Training data quality critically shapes model predictions, yet the causal relation between data points and outcomes remains opaque in deep models. This survey consolidates seven influential pointwise training-data-influence estimators, split into retraining-based and gradient-based families, and analyzes their definitions, assumptions, and computational trade-offs. It also surveys extensions, applications, and future directions, highlighting challenges in scalability, robustness, and group-level influence. The work provides a resource hub and a roadmap for developing more reliable, empirically grounded data-valuations.
Abstract
Good models require good training data. For overparameterized deep models, the causal relationship between training data and model predictions is increasingly opaque and poorly understood. Influence analysis partially demystifies training's underlying interactions by quantifying the amount each training instance alters the final model. Measuring the training data's influence exactly can be provably hard in the worst case; this has led to the development and use of influence estimators, which only approximate the true influence. This paper provides the first comprehensive survey of training data influence analysis and estimation. We begin by formalizing the various, and in places orthogonal, definitions of training data influence. We then organize state-of-the-art influence analysis methods into a taxonomy; we describe each of these methods in detail and compare their underlying assumptions, asymptotic complexities, and overall strengths and weaknesses. Finally, we propose future research directions to make influence analysis more useful in practice as well as more theoretically and empirically sound. A curated, up-to-date list of resources related to influence analysis is available at https://github.com/ZaydH/influence_analysis_papers.