An Efficient Framework for Crediting Data Contributors of Diffusion Models

TL;DR

The paper tackles fair attribution of global properties in diffusion models to data contributors by adopting Shapley-value theory. It introduces sparsified fine-tuning to dramatically reduce the computational burden of retraining and inference required for Shapley estimation, enabling practical credit assignment across contributor groups. Empirical results on CIFAR-20, CelebA-HQ, and ArtBench demonstrate that the proposed sparsified-ft Shapley method outperforms baselines in capturing contributors’ impact on global properties such as image quality, demographic diversity, and aesthetics, while achieving substantial speedups. The work has practical implications for data-sharing incentives, compensation policies, and fairness in diffusion-model training, with potential extensions to unlearning and larger models. Overall, the framework provides a scalable and principled approach to quantify data-contributor value for diffusion models, grounded in axiomatic fairness and supported by empirical validation.

Abstract

As diffusion models are deployed in real-world settings, and their performance is driven by training data, appraising the contribution of data contributors is crucial to creating incentives for sharing quality data and to implementing policies for data compensation. Depending on the use case, model performance corresponds to various global properties of the distribution learned by a diffusion model (e.g., overall aesthetic quality). Hence, here we address the problem of attributing global properties of diffusion models to data contributors. The Shapley value provides a principled approach to valuation by uniquely satisfying game-theoretic axioms of fairness. However, estimating Shapley values for diffusion models is computationally impractical because it requires retraining on many training data subsets corresponding to different contributors and rerunning inference. We introduce a method to efficiently retrain and rerun inference for Shapley value estimation, by leveraging model pruning and fine-tuning. We evaluate the utility of our method with three use cases: (i) image quality for a DDPM trained on a CIFAR dataset, (ii) demographic diversity for an LDM trained on CelebA-HQ, and (iii) aesthetic quality for a Stable Diffusion model LoRA-finetuned on Post-Impressionist artworks. Our results empirically demonstrate that our framework can identify important data contributors across models' global properties, outperforming existing attribution methods for diffusion models.

Figures (19)

• Figure 1: Schematic overview of our proposed framework, where $\theta^*$ denotes a trained diffusion model for which we aim to credit data contributors, and $\Tilde{\theta}^*$ denotes the pruned model that approximates $\theta^*$. After fine-tuning the pruned model on data corresponding to various subsets of contributors, denoted as $\Tilde{\theta}^{\text{ft}}$, and rerunning inference; global model properties ($\mathcal{F}$) are measured to estimate the Shapley value for each data contributor.
• Figure 2: Comparison of LDS (%) with $\alpha = 0.5$ among Shapley values estimated with sparsified fine-tuning (FT), fine-tuning (FT), and retraining under the same computational budgets (1 unit $=$ runtime to retrain and run inference on a full model). Specific runtimes are shown in \ref{['tab:results_computation_time']}.
• Figure 3: Relative percentage changes in global properties, comparing the original fully trained models to models retrained after removing the top 40% contributors (top) and including only the top contributors (bottom), as identified by various attribution methods.
• Figure 4: Top contributors and their corresponding training images for each dataset (top). Pairs of generated images above and below the 90th percentile of aesthetic score from Stable Diffusion models LoRA-finetuned under three conditions: excluding the data from the top 40% of artists, using the data from all the artists, and including only the data from the top 60% of artists (bottom).
• Figure 5: Pearson correlation between model behaviors evaluated with retraining vs. sparsified fine-tuning, with varying number of fine-tuning steps. Models are retrained or fine-tuned on 100 contributor subsets sampled from the Shapley kernel.

Theorems & Definitions (10)

• Definition 1
• Proposition 1
• Proposition 2
• Definition 2
• Lemma 1
• proof
• proof : Proof of \ref{['prop:model_behavior_approx']}
• Lemma 2
• proof
• proof : Proof of \ref{['cor:shapley_value_approx']}