Enhancing Diffusion-Based Quantitatively Controllable Image Generation via Matrix-Form EDM and Adaptive Vicinal Training

TL;DR

iCCDM addresses the limitations of prior continuous conditional diffusion by extending the Elucidated Diffusion Model to a matrix-form diffusion framework with condition-aware forward and reverse SDEs and a PF-ODE for efficient sampling. It introduces a nonnegative conditioning weight $\\lambda_y$ and a long embedding $\\bm{h}^l(y)$ transformed to a positive form, enabling flexible conditioning, plus a lightweight CNN covariance-embedding network to reduce memory. The approach combines adaptive vicinal training, matrix-form preconditioning, and second-order matrix-form sampling to achieve higher generation quality with substantially lower sampling cost than existing diffusion and GAN-based CCGM methods. Experiments on RC-49, UTKFace, Steering Angle, and Cell-200 across $64\times64$ to $256\times256$ resolutions demonstrate state-of-the-art SFID/NIQE/Diversity/LabelScore with notable speedups, confirming practical impact for quantitatively controllable image synthesis. The work provides a scalable, conditioning-aware diffusion framework for continuous attributes, enabling more reliable and efficient regression-conditioned image generation.

Abstract

Continuous Conditional Diffusion Model (CCDM) is a diffusion-based framework designed to generate high-quality images conditioned on continuous regression labels. Although CCDM has demonstrated clear advantages over prior approaches across a range of datasets, it still exhibits notable limitations and has recently been surpassed by a GAN-based method, namely CcGAN-AVAR. These limitations mainly arise from its reliance on an outdated diffusion framework and its low sampling efficiency due to long sampling trajectories. To address these issues, we propose an improved CCDM framework, termed iCCDM, which incorporates the more advanced \textit{Elucidated Diffusion Model} (EDM) framework with substantial modifications to improve both generation quality and sampling efficiency. Specifically, iCCDM introduces a novel matrix-form EDM formulation together with an adaptive vicinal training strategy. Extensive experiments on four benchmark datasets, spanning image resolutions from $64\times64$ to $256\times256$, demonstrate that iCCDM consistently outperforms existing methods, including state-of-the-art large-scale text-to-image diffusion models (e.g., Stable Diffusion 3, FLUX.1, and Qwen-Image), achieving higher generation quality while significantly reducing sampling cost.

Figures (3)

• Figure 1: Comparison of Sliding FID versus Sampling Speed Across Three Model Families on the Steering Angle Dataset ($64\times64$). The size of each scatter point represents GPU memory usage during sampling. Arrows ($\downarrow$ or $\uparrow$) indicate whether lower or higher values are preferred. Text-to-image diffusion models are fine-tuned from officially released checkpoints using either full fine-tuning or LoRA hu2022lora, while all other methods are trained from scratch.
• Figure 2: An Illustrative Example Highlighting the Superiority of iCCDM over State-of-the-Art Text-to-Image Models for Rotation-Angle Conditioning on RC-49.
• Figure 3: Effect of the Weighting Coefficient $\lambda_y$ in Eq. \ref{['eq:g_i']} on iCCDM Across Two $64\times64$ Datasets. $\lambda_y^t$ and $\lambda_y^s$ denote the values of $\lambda_y$ used during training and sampling, respectively.

Theorems & Definitions (8)

• Theorem 1
• Remark 1
• Remark 2
• Remark 3
• Remark 4
• Theorem S.2
• proof
• Corollary S.1: Nosing Perturbation