Inference-Time Loss-Guided Colour Preservation in Diffusion Sampling

TL;DR

This work presents an inference-time, region-constrained color preservation method that steers a pretrained diffusion model without any additional training, and shows that mean-only baselines can satisfy average color constraints while producing perceptually salient local failures, motivating the distribution-aware objective.

Abstract

Precise color control remains a persistent failure mode in text-to-image diffusion systems, particularly in design-oriented workflows where outputs must satisfy explicit, user-specified color targets. We present an inference-time, region-constrained color preservation method that steers a pretrained diffusion model without any additional training. Our approach combines (i) ROI-based inpainting for spatial selectivity, (ii) background-latent re-imposition to prevent color drift outside the ROI, and (iii) latent nudging via gradient guidance using a composite loss defined in CIE Lab and linear RGB. The loss is constructed to control not only the mean ROI color but also the tail of the pixelwise error distribution through CVaR-style and soft-maximum penalties, with a late-start gate and a time-dependent schedule to stabilize guidance across denoising steps. We show that mean-only baselines can satisfy average color constraints while producing perceptually salient local failures, motivating our distribution-aware objective. The resulting method provides a practical, training-free mechanism for targeted color adherence that can be integrated into standard Stable Diffusion inpainting pipelines.

Figures (12)

• Figure 1: Conversion pipeline. Overview of the differentiable colour conversion and loss computation flow.
• Figure 2: Loss function flow. Diagram illustrating the inference-time optimization process and data flow.
• Figure 3: Background Consistency. Notice the solid background colour when the latent is reinforced (left) versus the textured, inconsistent background when it is not (right).
• Figure 4: Qualitative Results and Error Heatmaps. Top row: Final image vs. target swatch, $\Delta E_{00}$ heatmap, and overlay. Bottom row: Component difference maps for $\Delta L$, $\Delta a$, and $\Delta b$.
• Figure 5: Loss and Optimization Dynamics. Evolution of total loss, component losses (LinearRGB vs. CVaR), gradient norms, and sRGB trajectory over 80 sampling steps.