DPAC: Distribution-Preserving Adversarial Control for Diffusion Sampling

TL;DR

This work tackles the instability of gradient-guided diffusion sampling by decomposing guidance into density-preserving (tangential) and density-changing (normal) components. By projecting adversarial gradients onto the tangent space defined by the score geometry, DPAC minimizes path-space KL and yields tighter bounds on $W_2$ and $FID$, while maintaining target attainment. The authors provide a rigorous theoretical framework (Girsanov, weighted Hodge decomposition, discrete bounds) and demonstrate practical gains on ImageNet-100, achieving lower FID and reduced energy consumption with a robust denoise-then-perturb injection. The approach is shown to be second-order robust to score/metric approximations and architecture-agnostic, offering a principled path to high-fidelity, adversarial diffusion guidance. Overall, DPAC unifies attack effectiveness with perceptual fidelity by an energy-minimization principle and practical tangential projection, delivering stable, efficient UAE generation.

Abstract

Adversarially guided diffusion sampling often achieves the target class, but sample quality degrades as deviations between the adversarially controlled and nominal trajectories accumulate. We formalize this degradation as a path-space Kullback-Leibler divergence(path-KL) between controlled and nominal (uncontrolled) diffusion processes, thereby showing via Girsanov's theorem that it exactly equals the control energy. Building on this stochastic optimal control (SOC) view, we theoretically establish that minimizing this path-KL simultaneously tightens upper bounds on both the 2-Wasserstein distance and Fréchet Inception Distance (FID), revealing a principled connection between adversarial control energy and perceptual fidelity. From a variational perspective, we derive a first-order optimality condition for the control: among all directions that yield the same classification gain, the component tangent to iso-(log-)density surfaces (i.e., orthogonal to the score) minimizes path-KL, whereas the normal component directly increases distributional drift. This leads to DPAC (Distribution-Preserving Adversarial Control), a diffusion guidance rule that projects adversarial gradients onto the tangent space defined by the generative score geometry. We further show that in discrete solvers, the tangent projection cancels the O(Δt) leading error term in the Wasserstein distance, achieving an O(Δt^2) quality gap; moreover, it remains second-order robust to score or metric approximation. Empirical studies on ImageNet-100 validate the theoretical predictions, confirming that DPAC achieves lower FID and estimated path-KL at matched attack success rates.

Figures (5)

• Figure 1: Toy 1D diffusion illustrating the link between path--KL and distribution-preserving control. Shown are three panels from left to right. Left: samples from uncontrolled ($P^0$) and controlled ($P^u$) 1D diffusions with $x_0\!\sim\!\mathcal{N}(0,0.5^2)$, $\mathrm{d}x_t=\log(t)\,\mathrm{d}t+\mathrm{d}W_t$ (uncontrolled), and $\mathrm{d}x_t=[\log(t)-2\log(t)]\,\mathrm{d}t+\mathrm{d}W_t$ (controlled, $u_t=-2\log t$). Thin lines show sample trajectories and thick lines their means, indicating that control shifts the mean path while keeping stochastic spread. Center: cumulative empirical $\mathrm{KL}(P^u\!\Vert\!P^0)$ (blue) and control energy $\tfrac12\sum_k u(t_k)^2\Delta t$ (orange) coincide, verifying $\mathrm{KL}(P^u\!\Vert\!P^0) =\tfrac12\mathbb{E}_{P^u}\!\int_0^1\!\|u_t\|^2\mathrm{d}t$ (Girsanov identity; final values 3.90, 3.92, 3.87). Right: iso-density contours of a 2D Gaussian mixture showing normal control (red, $u_t^{\mathrm{nor}}\!\propto\!\nabla\log p_t$) that changes density versus tangential control (green, $u_t^{\mathrm{tan}}$) that moves along level sets. Together, these panels visualize that reducing path--KL (control energy) yields density-preserving, tangential dynamics.
• Figure 2: Quantitative validation of DPAC on ImageNet-100 (200 steps).(a) Stability: AdvDiff (red) suffers catastrophic FID collapse (39.9 $\to$ 69.37) at high guidance scales ($\eta_k$). DPAC (blue) remains robustly stable. (b) Effectiveness & Efficiency: A direct comparison of the best FID each method achieved. DPAC (blue) achieves a superior peak fidelity (FID 33.90) while using only one-third of the energy ($\mathcal{E}$=54.0) that AdvDiff (red) required for its worse optimum (FID 34.66 at $\mathcal{E}$=160.0). (c) Theoretical Validation: At all scales, DPAC consistently uses $\approx$66% less energy, empirically validating our theory.
• Figure 3: Stability Analysis at High Scale ($\eta=10.0$). While SOTA Base (Red) suffers from rapid FID degradation (collapse) as the number of steps increases, DPAC (Blue) effectively bounds the error, validating the theoretical stability.
• Figure 4: Visual Analysis of Catastrophic Collapse. Comparison of samples generated under standard ($\eta=5.0$) vs. extreme ($\eta=10.0, N=200$) conditions. We display two sets of comparisons side-by-side. Standard (Ref): Clean samples generated with conservative settings. SOTA Base: At high guidance, the image collapses into severe noise and artifacts due to off-manifold drift. DPAC: While the extreme guidance forces a semantic shift from the original trajectory, DPAC effectively defends against artifact accumulation, producing natural and structurally coherent images unlike the baseline.
• Figure 5: Qualitative Comparison in Standard Regime ($\eta=2.5$). We display generated samples from the Baseline and DPAC across multiple classes. Compared to the unguided baseline (Clean), DPAC produces high-fidelity images that are visually indistinguishable, demonstrating that our method successfully attacks the classifier without compromising the perceptual quality of the original diffusion process.