DP-GENG : Differentially Private Dataset Distillation Guided by DP-Generated Data

TL;DR

This work tackles privacy risks in dataset distillation by introducing DP-GenG, a framework that uses DP-generated data to guide the distillation process under formal differential privacy. It leverages DP-generated data for initialization, feature extractor training, and a dedicated expert model to calibrate distilled examples, all while carefully allocating the privacy budget across components via GDP-based accounting. The approach yields significant gains in utility and robustness against membership inference attacks compared with state-of-the-art DP-DD methods, demonstrated on CIFAR-10/100 and CelebA under tight privacy budgets. By tightly integrating DP-generated data with distillation, DP-GenG advances a practical, privacy-preserving path for creating realistic, label-consistent, and useful distilled datasets. This work thus establishes a new paradigm for trustworthy dataset distillation that preserves privacy without sacrificing utility to the extent of prior DP-DD methods.

Abstract

Dataset distillation (DD) compresses large datasets into smaller ones while preserving the performance of models trained on them. Although DD is often assumed to enhance data privacy by aggregating over individual examples, recent studies reveal that standard DD can still leak sensitive information from the original dataset due to the lack of formal privacy guarantees. Existing differentially private (DP)-DD methods attempt to mitigate this risk by injecting noise into the distillation process. However, they often fail to fully leverage the original dataset, resulting in degraded realism and utility. This paper introduces \libn, a novel framework that addresses the key limitations of current DP-DD by leveraging DP-generated data. Specifically, \lib initializes the distilled dataset with DP-generated data to enhance realism. Then, generated data refines the DP-feature matching technique to distill the original dataset under a small privacy budget, and trains an expert model to align the distilled examples with their class distribution. Furthermore, we design a privacy budget allocation strategy to determine budget consumption across DP components and provide a theoretical analysis of the overall privacy guarantees. Extensive experiments show that \lib significantly outperforms state-of-the-art DP-DD methods in terms of both dataset utility and robustness against membership inference attacks, establishing a new paradigm for privacy-preserving dataset distillation.

Figures (9)

• Figure 1: We compare different dataset distillation methods in terms of the privacy, utility and realism of the resulting distilled datasets.
• Figure 2: The overall framework of DP-GenG . It fully leverages DP-generated data throughout the distillation process to enhance the performance of the distilled dataset. The blue, green and orange datasets represent the original private dataset, its DP-generated version and its distilled dataset (with DP guarantees), respectively.
• Figure 3: PSG, CelebA, $\texttt{IPC}\xspace=10$.
• Figure 4: NDPDC, CelebA, $\texttt{IPC}\xspace=10$.
• Figure 5: DP-GenG , CelebA, $\texttt{IPC}\xspace=10$.

Theorems & Definitions (11)

• Lemma 1: Gaussian Mechanism to GDP dong2022gaussian
• Theorem 1: Post-processing dwork2014algorithmic
• Lemma 2: GDP Composition dong2022gaussian
• Lemma 3: GDP to DP Conversion dong2022gaussian
• Theorem 2: DP-GenG Privacy Budget Allocation
• Definition 1: Differential Privacy dwork2006calibrating
• Definition 2: $f$-DP and $\mu$-Gaussian DP dong2022gaussian
• Theorem 3: DP-GENG Privacy Budget Allocation
• proof
• Lemma 4: Parallel Composition for GDP smith2021making