From Easy to Hard++: Promoting Differentially Private Image Synthesis Through Spatial-Frequency Curriculum

TL;DR

FETA-Pro introduces a spatial-frequency curriculum for DP image synthesis that augments DP-SGD with frequency-domain training shortcuts. By extracting spatial features via central images and learning frequency features through an auxiliary GAN to generate frequency-aligned images, the method learns in a structured easy-to-hard sequence: spatial features -> frequency features -> full images. Across five sensitive datasets, FETA-Pro achieves higher fidelity and downstream utility than state-of-the-art baselines at $\varepsilon=1$, while maintaining DP through Rényi-DP accounting. The work demonstrates that frequency features complement spatial cues, that an auxiliary generator is effective for unifying different feature domains, and that careful privacy-budget allocation (placing more budget on frequency features and DP-SGD) yields substantial practical gains without public data. This approach advances DP image synthesis by leveraging multi-synthesizer strengths and a pipeline-generation perspective to overcome single-model limitations.

Abstract

To improve the quality of Differentially private (DP) synthetic images, most studies have focused on improving the core optimization techniques (e.g., DP-SGD). Recently, we have witnessed a paradigm shift that takes these techniques off the shelf and studies how to use them together to achieve the best results. One notable work is DP-FETA, which proposes using `central images' for `warming up' the DP training and then using traditional DP-SGD. Inspired by DP-FETA, we are curious whether there are other such tools we can use together with DP-SGD. We first observe that using `central images' mainly works for datasets where there are many samples that look similar. To handle scenarios where images could vary significantly, we propose FETA-Pro, which introduces frequency features as `training shortcuts.' The complexity of frequency features lies between that of spatial features (captured by `central images') and full images, allowing for a finer-grained curriculum for DP training. To incorporate these two types of shortcuts together, one challenge is to handle the training discrepancy between spatial and frequency features. To address it, we leverage the pipeline generation property of generative models (instead of having one model trained with multiple features/objectives, we can have multiple models working on different features, then feed the generated results from one model into another) and use a more flexible design. Specifically, FETA-Pro introduces an auxiliary generator to produce images aligned with noisy frequency features. Then, another model is trained with these images, together with spatial features and DP-SGD. Evaluated across five sensitive image datasets, FETA-Pro shows an average of 25.7% higher fidelity and 4.1% greater utility than the best-performing baseline, under a privacy budget $ε= 1$.

Figures (7)

• Figure 1: The framework of FETA-Pro. During warm-up, FETA-Pro extracts spatial features to train the synthesizer. Then, FETA-Pro introduces auxiliary generators to generate images aligning the noisy frequency features, and training synthesizers on these synthetic images. Then, we train the warmed-up synthesizers on sensitive images using DP-SGD.
• Figure 2: Sampled images generated leveraging frequency ('Freq.') and spatial ('Spat.') domain features under $\epsilon=1$.
• Figure 3: Synthetic Image Examples under $\epsilon = 10$. The last row of images is real image samples from sensitive image datasets. Due to space constraints, we showcase only the top-4 performing baseline methods: DP-Kernel, DP-GAN, DPDM, and DP-FETA.
• Figure 4: FID of synthetic images during fine-tuning, compared to baseline methods to evaluate convergence under $\epsilon=1$. The FID is computed using 6,000 sampled images (10% of the final synthetic dataset) relative to the sensitive dataset.
• Figure 5: FID (top row) and Acc (bottom row) of FETA-Pro and five baselines with $\varepsilon=1$. 'DPDM' indicates no warm-up. 'DP-FETA' and '$\text{FETA-Pro}_{f}$' use only spatial and frequency features for warm-up, respectively. '$\text{FETA-Pro}_{mix}$' learns spatial and frequency features simultaneously. '$\text{FETA-Pro}_{ft}$' first learns frequency domain features, then spatial features. 'FETA-Pro' is our work, which learns spatial domain features, then frequency features.