FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging

TL;DR

FastIF tackles the prohibitive runtime of influence functions by combining a kNN-based candidate reduction, a tuned inverse Hessian–vector product estimator, and parallelization. The approach yields roughly 80x speedups while preserving high fidelity to full influence values, enabling scalable interpretation and targeted error correction. Empirically, FastIF supports applications in explainability, visualization of influence interactions, and data-driven model improvement, including improvements on HANS and ANLI via influential-data augmentation. The work demonstrates that influence-based analysis can be practically applied to large models and datasets, unlocking reliable, data-point–level debugging and insight without retraining from scratch. It also provides a framework for future efficiency-focused explorations in influence-guided learning and evaluation.

Abstract

Influence functions approximate the "influences" of training data-points for test predictions and have a wide variety of applications. Despite the popularity, their computational cost does not scale well with model and training data size. We present FastIF, a set of simple modifications to influence functions that significantly improves their run-time. We use k-Nearest Neighbors (kNN) to narrow the search space down to a subset of good candidate data points, identify the configurations that best balance the speed-quality trade-off in estimating the inverse Hessian-vector product, and introduce a fast parallel variant. Our proposed method achieves about 80X speedup while being highly correlated with the original influence values. With the availability of the fast influence functions, we demonstrate their usefulness in four applications. First, we examine whether influential data-points can "explain" test time behavior using the framework of simulatability. Second, we visualize the influence interactions between training and test data-points. Third, we show that we can correct model errors by additional fine-tuning on certain influential data-points, improving the accuracy of a trained MultiNLI model by 2.5% on the HANS dataset. Finally, we experiment with a similar setup but fine-tuning on datapoints not seen during training, improving the model accuracy by 2.8% and 1.7% on HANS and ANLI datasets respectively. Overall, our fast influence functions can be efficiently applied to large models and datasets, and our experiments demonstrate the potential of influence functions in model interpretation and correcting model errors. Code is available at https://github.com/salesforce/fast-influence-functions

Figures (13)

• Figure 1: Workflow of FastIF w.r.t. a test data-point. First a subset of data-points are selected from the entire training set using $k$NN to reduce search space, then the inverse Hessian-vector product ($s_{\text{test}}$) is estimated based on Sec. \ref{['subsec:speedup-ihvp']}. The influence values of data-points are computed using the outputs from these two steps. Finally, the most influential data-point(s) are returned.
• Figure 2: Parallel computation of influence functions.
• Figure 3: Please see Appendix Figs. \ref{['appendix-figure:analysis-knn-recalls']},\ref{['appendix-figure:analysis-hessian-speed-quality-tradeoff']},\ref{['appendix-figure:retraining']} for higher resolution versions. (a) The recall of $k$NN in terms of finding influential data-points. The lines/error bars represent the means/standard deviations across $100$ correct/incorrect predictions. (b) Computational time (left) and estimation error norm (right) of Hessian approximation. (c) Change in loss on the data-point after retraining, where we remove $m_{\text{remove}} {\in} \{1, 5, 25, 50, 100\}$ data-points.
• Figure 4: Simulator loss on $4$ test data-points (more figures in the appendix), where the simulator is fine-tuned on different types of data-points with ground truth labels using various learning rates. The lines refer to the mean performance averaged across $10$ fine-tuning data-points, and the shaded area covers the max/min performance.
• Figure 5: Visualization of the interaction between evaluation and training data-points.