CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation

TL;DR

This paper investigates controllable and constrained sampling for diffusion models by focusing on how initial noise perturbations influence generated data under DDIM and its ODE formulation. It reveals a highly linear input-output relationship with respect to perturbation scale and exploits this through the Controllable and Constrained Sampling (CCS) framework, using spherical interpolation and a controller to achieve a target mean with adjustable diversity while maintaining sample quality. The approach is extended to latent diffusion models with Partial-Inversion CCS Sampling (P-CCS) to enable CFG-based editing. Extensive experiments across pixel and latent diffusion models demonstrate precise mean-centering, strong linearity in perturbation effects, and competitive or superior image quality and diversity compared with baselines, highlighting practical applicability to image editing and constrained-generation tasks.

Abstract

Diffusion models have emerged as powerful tools for generative tasks, producing high-quality outputs across diverse domains. However, how the generated data responds to the initial noise perturbation in diffusion models remains under-explored, which hinders understanding the controllability of the sampling process. In this work, we first observe an interesting phenomenon: the relationship between the change of generation outputs and the scale of initial noise perturbation is highly linear through the diffusion ODE sampling. Then we provide both theoretical and empirical study to justify this linearity property of this input-output (noise-generation data) relationship. Inspired by these new insights, we propose a novel Controllable and Constrained Sampling method (CCS) together with a new controller algorithm for diffusion models to sample with desired statistical properties while preserving good sample quality. We perform extensive experiments to compare our proposed sampling approach with other methods on both sampling controllability and sampled data quality. Results show that our CCS method achieves more precisely controlled sampling while maintaining superior sample quality and diversity.

Figures (9)

• Figure 1: CCS Sampling with a target mean in the FFHQ validation dataset with $C_0 = 0.4$
• Figure 2: Qualitative demonstration of linearity when increasing scale of perturbation. For each target mean, we sample a perturbation noise and gradually increase $C_0$ (0.1 at a time) to increase the magnitude of the perturbation.
• Figure 3: Quantitative demonstration of linearity when increasing scale of perturbation. With increased $\sin(C_0)$, the magnitude of perturbation increases, and the average $L^2$ distance between samples and the target image increases linearly. Left is the linearity on FFHQ dataset using pixel diffusion; Right is the linearity on Celeba-HQ dataset using Stable Diffusion 1.5.
• Figure 4: We sample 120 images with a fixed target mean using different methods and analyze their sample mean (average pixel intensity). Our observations show that the sample mean of our method closely matches that of the original image.
• Figure 5: Example Samples of P-CCS around an Edited Mean.

Theorems & Definitions (11)

• Proposition 1
• Proposition 2
• Proposition 3
• Proposition 4
• Proposition 5
• Proposition 6
• proof : Proof of Proposition \ref{['prop: linearity, DDIM']}
• proof : Proof of Proposition \ref{['prop: linearity ODE']}
• proof : Proof of Proposition \ref{['prop: at most linear ODE']}
• proof : Proof of Proposition \ref{['prop: centering feasibility']}