Nonparametric Data Attribution for Diffusion Models

TL;DR

This work tackles data attribution for diffusion models without requiring gradient access or retraining. It introduces Nonparametric Diffusion Attribution (NDA), which computes patch-level influence by comparing local image patches against training patches under the analytically derived optimal score framework, extended to multiscale representations and accelerated via convolutions. NDA achieves strong alignment with ground-truth influence measured by LDS and competitive performance against gradient-based methods, while providing spatial interpretability through patch-level localization. The method demonstrates robust, model-agnostic attribution across CIFAR-2, CIFAR-10, and CelebA, enabling practical data transparency and accountability for large-scale diffusion pipelines.

Abstract

Data attribution for generative models seeks to quantify the influence of individual training examples on model outputs. Existing methods for diffusion models typically require access to model gradients or retraining, limiting their applicability in proprietary or large-scale settings. We propose a nonparametric attribution method that operates entirely on data, measuring influence via patch-level similarity between generated and training images. Our approach is grounded in the analytical form of the optimal score function and naturally extends to multiscale representations, while remaining computationally efficient through convolution-based acceleration. In addition to producing spatially interpretable attributions, our framework uncovers patterns that reflect intrinsic relationships between training data and outputs, independent of any specific model. Experiments demonstrate that our method achieves strong attribution performance, closely matching gradient-based approaches and substantially outperforming existing nonparametric baselines. Code is available at https://github.com/sail-sg/NDA.

Figures (14)

• Figure 1: Schematic illustration of attribution methods. Left: Model-based attribution relies on gradients and requires parameter access. Right: Our Nonparametric Diffusion Attribution (NDA) compares local image patches via an analytic score function, enabling attribution without model access.
• Figure 2: Counterfactual evaluation of $\ell_2$ distance (Left) and CLIP similarity (Right) between original and regenerated images on CIFAR-2 and CelebA after removing the most influential training samples identified by different attribution methods and retraining the model.
• Figure 3: Counterfactual visualization on CIFAR-2 (Top) and CelebA (Bottom). Images are compared to those generated by retrained models using the same seed. See Appendix \ref{['appendix:vis_counter']} for more cases.
• Figure 4: Spatial interpretability of NDA. The patches with the highest patch-wise influence scores (w.r.t. patches in the generated image) are highlighted in the top attributed training images.
• Figure 5: LDS (%) on the CIFAR-2 validation set across different patch sizes and timesteps at the original resolution. Left: small timesteps ($t{\leq}300$), where moderate patch sizes achieve the highest scores. Middle: medium timesteps ($t{=}400,500$), where larger patches outperform smaller ones due to higher noise. Right: large timesteps, where noise dominates and informative signal is minimal.