Self-Guided Generation of Minority Samples Using Diffusion Models

TL;DR

This work tackles generating minority samples that inhabit low-density regions of the data distribution by proposing a self-contained diffusion-based sampler that requires only a pretrained model. It introduces an inference-time minority metric derived from the posterior mean via Tweedie’s formula and couples it with a gradient-based self-guidance, augmented by intermittent and variance-based time-scheduling to improve minority-feature fidelity without external classifiers. Theoretical connections show the metric approximates the negative ELBO, linking minority guidance to likelihood objectives, while empirical results on CelebA, LSUN-Bedrooms, and ImageNet demonstrate sharp gains in low-density metrics and sample quality with practical inference costs. The approach offers strong practical advantages for data augmentation, fairness, and creative AI, and is extensible to T2I, medical imaging, and editing tasks using the same pretrained diffusion backbone.

Abstract

We present a novel approach for generating minority samples that live on low-density regions of a data manifold. Our framework is built upon diffusion models, leveraging the principle of guided sampling that incorporates an arbitrary energy-based guidance during inference time. The key defining feature of our sampler lies in its \emph{self-contained} nature, \ie, implementable solely with a pretrained model. This distinguishes our sampler from existing techniques that require expensive additional components (like external classifiers) for minority generation. Specifically, we first estimate the likelihood of features within an intermediate latent sample by evaluating a reconstruction loss w.r.t. its posterior mean. The generation then proceeds with the minimization of the estimated likelihood, thereby encouraging the emergence of minority features in the latent samples of subsequent timesteps. To further improve the performance of our sampler, we provide several time-scheduling techniques that properly manage the influence of guidance over inference steps. Experiments on benchmark real datasets demonstrate that our approach can greatly improve the capability of creating realistic low-likelihood minority instances over the existing techniques without the reliance on costly additional elements. Code is available at \url{https://github.com/soobin-um/sg-minority}.

Figures (24)

• Figure 1: (Left) Existing methods vs. our self-guided approach. Unlike previous methods that rely upon external components (e.g., classifiers) to guide the generation process towards low-density regions sehwag2022generatingum2023don, our approach yields low-density guidance solely based on a pretrained diffusion model, thereby offering a self-contained minority generation achievable without any aids of expensive extra elements. (Right) Overview of our self guidance for minority data. Specifically to yield low-density guidance given a current latent instance ${\boldsymbol x}_t$ during inference, we first obtain its denoised version $\hat{{\boldsymbol x}}_0$ via the use of Tweedie's formula robbins1992empiricalchung2022diffusion implemented with a pretrained model. We then perturb $\hat{{\boldsymbol x}}_0$ into $\hat{{\boldsymbol x}}_s$ via the DDPM forward process and denoise $\hat{{\boldsymbol x}}_s$ to $\hbox{$\hat{\space}$}{\hat{\boldsymbol x}}_0$ via the pretrained model. A discrepancy between $\hat{{\boldsymbol x}}_0$ and $\hbox{$\hat{\space}$}{\hat{\boldsymbol x}}_0$ (denoted as $\tilde{{\cal L}}({\boldsymbol x}_t, s)$ in the figure) is then computed, and we subsequently use its gradient as low-density guidance with the stopgrad technique chen2021exploring applied on $\hbox{$\hat{\space}$}{\hat{\boldsymbol x}}_0$; see \ref{['sec:method']} for details.
• Figure 2: Effectiveness of our metric (i.e., \ref{['eq:mse']}) for identifying low-likelihood minority features during inference time. Generated samples with the smallest (left column), moderate (middle column), and the highest (right column) values of the proposed metric are exhibited. We employed ADM dhariwal2021diffusion for the generations of all three benchmarks. The metric values were calculated during inference time via \ref{['eq:mse']}.
• Figure 3: Sample comparison on LSUN-Bedrooms. We share the same random seed for all methods.
• Figure 4: Sample comparison on ImageNet-256. Generated samples from two classes are exhibited: Water tower (top row) and Bald eagle (bottom row). For each row, we share the same random seed across all three methods.
• Figure 5: Comparison of neighborhood density on LSUN-Bedrooms. “AvgkNN” refers to Average k-Nearest Neighbor, and “LOF” is Local Outlier Factor breunig2000lof. "Rarity Score" indicates a low-density metric proposed by han2022rarity. The higher values, the less likely samples for all three measures.

Theorems & Definitions (6)

• proposition thmcounterproposition
• corollary thmcountercorollary
• Proposition \ref{proposition}
• proof
• Corollary \ref{corollary}
• proof