Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data Unlearning Beyond Uniform Forgetting via Diffusion Time and Frequency Selection

Jinseong Park, Mijung Park

TL;DR

This work tackles data unlearning in diffusion models, identifying that forgetting is not uniform across diffusion time steps and frequency bands. It introduces a time-frequency selective unlearning framework that applies a nonuniform time-step weighting and FFT-based low-pass filtering to target forgetting to semantic content while preserving detail, formalized as learning over a low-pass transformed state $\tilde{\mathbf{x}}_t=\mathcal{F}(\mathbf{x}_t)$ with a nonuniform distribution $P(t)$. The approach is compatible with existing unlearning objectives (e.g., GA, SISS, DPO, KTO) and is validated on image-level CelebA-HQ and text-to-image Stable Diffusion tasks, achieving higher aesthetic quality and stronger deletion signals via a normalized SSCD metric ($\mathrm{SSCD}^{\text{norm}}$) with only modest changes to retained-sample quality. A normalized SSCD metric is proposed to jointly assess deletion and sample quality, and the study discusses limitations and future directions for automatic discovery of optimal unlearning regions.

Abstract

Data unlearning aims to remove the influence of specific training samples from a trained model without requiring full retraining. Unlike concept unlearning, data unlearning in diffusion models remains underexplored and often suffers from quality degradation or incomplete forgetting. To address this, we first observe that most existing methods attempt to unlearn the samples at all diffusion time steps equally, leading to poor-quality generation. We argue that forgetting occurs disproportionately across time and frequency, depending on the model and scenarios. By selectively focusing on specific time-frequency ranges during training, we achieve samples with higher aesthetic quality and lower noise. We validate this improvement by applying our time-frequency selective approach to diverse settings, including gradient-based and preference optimization objectives, as well as both image-level and text-to-image tasks. Finally, to evaluate both deletion and quality of unlearned data samples, we propose a simple normalized version of SSCD. Together, our analysis and methods establish a clearer understanding of the unique challenges in data unlearning for diffusion models, providing practical strategies to improve both evaluation and unlearning performance.

Data Unlearning Beyond Uniform Forgetting via Diffusion Time and Frequency Selection

TL;DR

This work tackles data unlearning in diffusion models, identifying that forgetting is not uniform across diffusion time steps and frequency bands. It introduces a time-frequency selective unlearning framework that applies a nonuniform time-step weighting and FFT-based low-pass filtering to target forgetting to semantic content while preserving detail, formalized as learning over a low-pass transformed state with a nonuniform distribution . The approach is compatible with existing unlearning objectives (e.g., GA, SISS, DPO, KTO) and is validated on image-level CelebA-HQ and text-to-image Stable Diffusion tasks, achieving higher aesthetic quality and stronger deletion signals via a normalized SSCD metric () with only modest changes to retained-sample quality. A normalized SSCD metric is proposed to jointly assess deletion and sample quality, and the study discusses limitations and future directions for automatic discovery of optimal unlearning regions.

Abstract

Data unlearning aims to remove the influence of specific training samples from a trained model without requiring full retraining. Unlike concept unlearning, data unlearning in diffusion models remains underexplored and often suffers from quality degradation or incomplete forgetting. To address this, we first observe that most existing methods attempt to unlearn the samples at all diffusion time steps equally, leading to poor-quality generation. We argue that forgetting occurs disproportionately across time and frequency, depending on the model and scenarios. By selectively focusing on specific time-frequency ranges during training, we achieve samples with higher aesthetic quality and lower noise. We validate this improvement by applying our time-frequency selective approach to diverse settings, including gradient-based and preference optimization objectives, as well as both image-level and text-to-image tasks. Finally, to evaluate both deletion and quality of unlearned data samples, we propose a simple normalized version of SSCD. Together, our analysis and methods establish a clearer understanding of the unique challenges in data unlearning for diffusion models, providing practical strategies to improve both evaluation and unlearning performance.
Paper Structure (28 sections, 13 equations, 24 figures, 1 table)

This paper contains 28 sections, 13 equations, 24 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Diffusion Models
  4. Data Unlearning in Diffusion Models
  5. Machine Unlearning.
  6. Diffusion Unlearning.
  7. Quality Degradation in Unlearning.
  8. Selective Data Unlearning in Diffusion Models
  9. Important Features in Diffusion Unlearning
  10. Hypothesis 1: Unlearning the specific diffusion stage is enough.
  11. Hypothesis 2: High-frequency components are not necessarily unlearned.
  12. Selective Unlearning with Time Step Selection and Frequency Filtering
  13. Experiments
  14. Experimental Setups
  15. Datasets and Baselines.
  16. ...and 13 more sections

Figures (24)

  • Figure 1: Illustration of quality degradation of unlearned images after data unlearning. Letf: two images to unlearn. Middle: the results of Gradient Ascent (GA, first column) and those of GA together with our approach (second column). Right: the results of SISS (first column) and those of SISS together with our approach (second column). Our approach, combined with existing unlearning objective functions, improves the quality of generated unlearned samples.
  • Figure 2: Illustration of unlearning the specific stage of time steps only in diffusion models.
  • Figure 3: Analysis on early refinement stage.
  • Figure 4: Analysis on middle-late stage.
  • Figure 5: Power Spectral Density (PSD) before and after unlearning. For collapsed forget data, the difference in PSD is significant in high-frequency regions.
  • ...and 19 more figures