Stochastic Conditional Diffusion Models for Robust Semantic Image Synthesis

TL;DR

This work tackles semantic image synthesis under noisy user inputs by proposing Stochastic Conditional Diffusion Model (SCDM), which couples a discrete forward diffusion for labels (Label Diffusion) with a continuous reverse diffusion for images. It introduces a class-aware noise schedule and an absorbing-state mechanism to align intermediate conditioning across noisy and clean inputs, improving robustness and semantic fidelity. The method is validated on multiple datasets, including a new noisy-SIS benchmark, and demonstrates strong performance in FID, LPIPS, and mIoU compared to GAN- and diffusion-based baselines, while maintaining diversity and realism. The results support practical applicability of diffusion-based SIS in real-world interactive scenarios and provide a publicly available implementation.

Abstract

Semantic image synthesis (SIS) is a task to generate realistic images corresponding to semantic maps (labels). However, in real-world applications, SIS often encounters noisy user inputs. To address this, we propose Stochastic Conditional Diffusion Model (SCDM), which is a robust conditional diffusion model that features novel forward and generation processes tailored for SIS with noisy labels. It enhances robustness by stochastically perturbing the semantic label maps through Label Diffusion, which diffuses the labels with discrete diffusion. Through the diffusion of labels, the noisy and clean semantic maps become similar as the timestep increases, eventually becoming identical at $t=T$. This facilitates the generation of an image close to a clean image, enabling robust generation. Furthermore, we propose a class-wise noise schedule to differentially diffuse the labels depending on the class. We demonstrate that the proposed method generates high-quality samples through extensive experiments and analyses on benchmark datasets, including a novel experimental setup simulating human errors during real-world applications. Code is available at https://github.com/mlvlab/SCDM.

Figures (21)

• Figure 1: Visualization of conditional generation. Each colored trajectory represents a sampling trajectory conditioned on a noisy semantic map $\tilde{\mathbf{y}}_0$(Red) and the corresponding clean semantic map $\mathbf{y}_0$(Blue). They are projected onto the (1) semantic map space and the (2) image space, sharing the same $\mathbf{x}_T$. (a) Existing conditional diffusion models (baseline) use a fixed condition $\mathbf{y}_0$ over the generation process, and the gap between $\tilde{\mathbf{y}}_t$ and $\mathbf{y}_{t}$ yields erroneous conditional score estimation at each timestep $t$. (b) In contrast, our method stochastically perturbs the condition with masking, resulting in a trajectory $\mathbf{y}_{1:T}$ following a probability distribution $q(\mathbf{y}_{1:T}|\mathbf{y}_0)$, as depicted with blue shaded areas around the $\mathbf{y}_{t}$ trajectory. This makes the intermediate trajectories, i.e., $\mathbf{y}_{1:T}|\mathbf{y}_0$ and $\tilde{\mathbf{y}}_{1:T}|\tilde{\mathbf{y}}_0$, close to each other, enhancing the robustness against the noisy labels.
• Figure 2: Generation process of SCDM. The Stochastic Conditional Diffusion Model (SCDM) is a robust conditional diffusion model for semantic image synthesis. SCDM consists of a discrete forward process for labels and a continuous reverse process for images. It improves the robustness to noisy semantic labels as well as generation performance on clean semantic labels. $\boldsymbol{z}_0$ denotes the $i$-th pixel of the semantic map, i.e., $\boldsymbol{z}_0 = \mathbf{y}_0^{i}$ where $\mathbf{y}_0 = \{ \mathbf{y}_0^{1},...,\mathbf{y}_0^{H \times W} \}$.
• Figure 3: Generation results on noisy labels.
• Figure 4: Generation results with and without Label Diffusion. The results are sampled with the fixed random seeds and the same $\mathbf{x}_T$, and generated with (a) clean labels, (b) DS, (c) Edge, and (d) Random setup noisy labels, respectively.
• Figure 5: Visualization of $\gamma_{t,c}$ throughout diffusion in the baseline, linear and uniform, and class-wise noise schedule.$\gamma_{t,c}$ indicates the probability of a label $c$ has transitioned to the absorbing state until timestep $t$. In our class-wise schedule, small and rare objects tend to be intact relatively longer in the diffusion process, e.g., clock.

Theorems & Definitions (6)

• Proposition 1
• Proposition 2
• Proposition 1
• proof
• Proposition 2
• proof