Z0-Inf: Zeroth Order Approximation for Data Influence

TL;DR

Zeroth-order Inferencing for Data Influence introduces Z0-Inf, a zeroth-order approximation for estimating training data influence that avoids gradient and inverse-Hessian computations. By fitting a finite-difference directional gradient between consecutive checkpoints and aggregating across epochs, the method uses only loss values and checkpoint states, making it applicable to non-differentiable losses and scalable to large models. The approach yields superior accuracy for self-influence and competitive performance for train-test influence on fine-tuned large language models, while delivering substantial runtime and memory savings through optimizations like dot product expansions and offline loss precomputation. It enables scalable data attribution, data pruning, and model debugging for modern neural networks, with clear guidance on when zeroth-order estimates align with SSRT baselines and where train-test estimates may be noisier. The work also discusses limitations and future directions, including group influence and alternative variance-based measures for self-influence.

Abstract

A critical aspect of analyzing and improving modern machine learning systems lies in understanding how individual training examples influence a model's predictive behavior. Estimating this influence enables critical applications, including data selection and model debugging; in particular, self-influence, which quantifies the influence of a training point on itself, has found many uses in data quality assessment and outlier detection. Existing methods for measuring data influence, however, are often impractical for large models due to low accuracy or prohibitive computational costs: most approaches either provide poor approximations or rely on gradients and inverse-Hessian computations that remain challenging to scale. In this work, we introduce a highly efficient zeroth-order approximation for estimating the influence of training data that requires only a fraction of the time and memory footprint of prior methods. Notably, our method relies solely on loss values of intermediate checkpoints on the training and test data, along with the checkpoints themselves, making it broadly applicable even when the loss function of interest is non-differentiable. Beyond its computational efficiency, our approach achieves superior accuracy in estimating self-influence and comparable or improved accuracy in estimating train-test influence for fine-tuned large language models, enabling scalable and practical analysis of how training data shapes model behavior.

Figures (6)

• Figure 1: The common subset of top-k self-influential examples between SSRT and four different influence approximation methods for three fine-tuned LLMs (on left) and non-LLMs (on right) in percentages.
• Figure 2: Average Spearman correlation between TracIn, Zeroth-Order, LoGra and SSRT for Top-5 and Top-10 most influential training examples. The training examples are selected based on the largest aggregate influence over all test examples for each method.
• Figure 3: Left: Runtime performance comparison across different optimization methods for six different models. The offline precompute required for our method is overlayed in dark gray. Right: Spearman rank correlation coefficient between SSRT, LoGra and First and Zeroth-order approximation methods.
• Figure 4: Top-1 Train-Test influence pairs for the CIFAR-10 dataset based on the SSRT, Zeroth and First order approximation methods.
• Figure 5: Top-3 train-test influence pairs for the CIFAR10 dataset based on the zeroth-order finite differences gradient approximation.