GUDA: Counterfactual Group-wise Training Data Attribution for Diffusion Models via Unlearning

TL;DR

GUDA presents a scalable approach to group-wise data attribution for diffusion models by approximating counterfactual LOGO models through machine unlearning starting from the full-data model. Attribution is measured via differences in the ELBO between the full model and unlearned counterfactuals, enabling efficient estimation of group influence. The framework provides two instantiations: Guda-U for unconditional generation using ReTrack-style redirection and Guda-C for conditional text-to-image using weighted style anchors, with empirical validation on CIFAR-10 and UnlearnCanvas showing superior head-focused performance and substantial speedups over LOGO. This work advances model transparency and accountability by offering practical, scalable group-level attribution, suitable for copyright, data-provider compensation, and debugging in large diffusion pipelines.

Abstract

Training-data attribution for vision generative models aims to identify which training data influenced a given output. While most methods score individual examples, practitioners often need group-level answers (e.g., artistic styles or object classes). Group-wise attribution is counterfactual: how would a model's behavior on a generated sample change if a group were absent from training? A natural realization of this counterfactual is Leave-One-Group-Out (LOGO) retraining, which retrains the model with each group removed; however, it becomes computationally prohibitive as the number of groups grows. We propose GUDA (Group Unlearning-based Data Attribution) for diffusion models, which approximates each counterfactual model by applying machine unlearning to a shared full-data model instead of training from scratch. GUDA quantifies group influence using differences in a likelihood-based scoring rule (ELBO) between the full model and each unlearned counterfactual. Experiments on CIFAR-10 and artistic style attribution with Stable Diffusion show that GUDA identifies primary contributing groups more reliably than semantic similarity, gradient-based attribution, and instance-level unlearning approaches, while achieving x100 speedup on CIFAR-10 over LOGO retraining.

Figures (5)

• Figure 1: Overview of the Guda (Group Unlearning-based Data Attribution) framework. Instead of retraining $N+1$ models from scratch (LOGO), Guda starts from the all-group model $\theta^{\mathrm{full}}$ and applies machine unlearning to obtain approximate counterfactual models $\theta^{\mathrm{ul}}_{-k}$ for each group $k$. Attribution is computed by comparing the ELBO of generated samples under the all-group model versus each unlearned model, providing an efficient approximation to the counterfactual target validated against LOGO.
• Figure 2: Qualitative comparison of group-wise attribution methods on UnlearnCanvas. For two generated images, we show the top-3 attributed styles for each method. Each row corresponds to: LOGOA (oracle target), Guda (Ours), CLIPA, and Wang et al.. Representative training images from each attributed style are displayed. Green boxes indicate agreement with the LOGOA top-3 styles.
• Figure 3: CIFAR-10 generations under group removal. Each panel pairs the all-class model with a counterfactual model trained without class 1 (automobile) or class 7 (horse). We use DDIM with 100 steps and the same initial noise for each pair. The top row shows LOGO models, and the bottom row shows ReTrack and ESD unlearned models.
• Figure 4: UnlearnCanvas training data examples for the Dogs class across the 16 evaluation styles. Images within the same style exhibit consistent visual characteristics, supporting style-level attribution analysis.
• Figure 5: Evaluation images generated by the all-group model using the evaluation prompts. Due to partial descriptor overlap across styles, some generations reflect a dominant style while others exhibit mixed visual characteristics.