Faithful and Fast Influence Function via Advanced Sampling

TL;DR

Influence functions provide a Hessian-based, post-hoc lens to quantify training-data influence on black-box models via the relation $\mathcal{I}(z) = - H_{\hat{\theta}}^{-1} \nabla_\theta l(z;\hat{\theta})$. To overcome costly Hessian inversions in large models, the paper introduces feature-based and logit-based advanced sampling to produce representative Hessian estimates with far fewer data points. Empirical results on class-removal tasks demonstrate that logit-based sampling yields the strongest accuracy and consistency, while delivering substantial efficiency gains (up to 30.1% reduction in runtime and 42.2% memory savings) and a roughly 2.5% boost in F1-score over baselines. Overall, the approach enables faithful, scalable post-hoc analysis and unlearning for large-scale, black-box models.

Abstract

How can we explain the influence of training data on black-box models? Influence functions (IFs) offer a post-hoc solution by utilizing gradients and Hessians. However, computing the Hessian for an entire dataset is resource-intensive, necessitating a feasible alternative. A common approach involves randomly sampling a small subset of the training data, but this method often results in highly inconsistent IF estimates due to the high variance in sample configurations. To address this, we propose two advanced sampling techniques based on features and logits. These samplers select a small yet representative subset of the entire dataset by considering the stochastic distribution of features or logits, thereby enhancing the accuracy of IF estimations. We validate our approach through class removal experiments, a typical application of IFs, using the F1-score to measure how effectively the model forgets the removed class while maintaining inference consistency on the remaining classes. Our method reduces computation time by 30.1% and memory usage by 42.2%, or improves the F1-score by 2.5% compared to the baseline.

Figures (3)

• Figure 1: Overview of our approach. A quick evaluation shows the performance of three sampling methods under five metrics: exclusive-loss (EL), self-loss (SL), F1 score (F1), run-time efficiency (RTE), and memory efficiency (ME). Results show that improved samplings lead to more accurate estimations of unlearning effects within less memory and time. The influence functions require Hessian matrix of the sampled training dataset and gradient vector of the target data. In conventional methods, the Hessian matrix is (a) intractable or (b) possibly unreliable. In our method, advanced samplers can choose a small but representative subset based on (c) feature and (d) logits.
• Figure 2: Evaluation results on the class removal task for VGG11 with CIFAR-10. Lower values indicate better performance, except for $\mathrm{F}_1$-score.
• Figure 3: Standard deviation of evaluation metrics presented in Fig. \ref{['fig:2']}.