Retinex-Diffusion: On Controlling Illumination Conditions in Diffusion Models via Retinex Theory

TL;DR

The paper presents Retinex-Diffusion, a training-free framework that controls illumination in diffusion models by guiding the reverse process with an illumination-energy term decomposed into illumination-variant and -invariant parts. It leverages a Retinex-inspired illumination extraction and a Gaussian illumination prompt to steer lighting, including shadows and inter-reflections, while preserving geometry in real-image relighting via CCR-based guidance. The method demonstrates realistic illumination generation and editing on generated and real images using pre-trained diffusion models, with ablations confirming the importance of illumination-property extraction and geometry preservation. This approach enables practical, plug-and-play illumination control without additional training or data labels, expanding diffusion models' applicability in photorealistic rendering and relighting tasks.

Abstract

This paper introduces a novel approach to illumination manipulation in diffusion models, addressing the gap in conditional image generation with a focus on lighting conditions. We conceptualize the diffusion model as a black-box image render and strategically decompose its energy function in alignment with the image formation model. Our method effectively separates and controls illumination-related properties during the generative process. It generates images with realistic illumination effects, including cast shadow, soft shadow, and inter-reflections. Remarkably, it achieves this without the necessity for learning intrinsic decomposition, finding directions in latent space, or undergoing additional training with new datasets.

Figures (10)

• Figure 1: Motivated by the recent state-of-the-art image manipulation method brooks2023instructpix2pix can not handle the low-level manipulation, such as relighting the scene, or creating illumination effect (top row). We present physics-guided and training-free diffusion for controlling illumination conditions in images. In image synthesis, our method generates photo-realistic illumination conditions under the proper illumination property guidance. Our model is also able to perform illumination editing of the original images, such as adding new illumination to images or face relighting (bottom row). Our approach is training-free and easily integrated with most pixel-based diffusion models, enhancing their illumination control capabilities efficiently.
• Figure 2: Overall diagram of the proposed illumination control diffusion method. Top, the illumination guidance image generation; Bottom, image relighting. $x_t$ represents the image at time step $t$, $\epsilon_\theta$ is the pre-trained U-Net unet. $y_s$ and $y_c$ are prompts for illumination guidance and the geometry persevering guidance.
• Figure 3: Illumination Property-Guided Image Generation: Each pair of columns displays the generated image alongside its corresponding illumination feature. The initial two columns show the original image without illumination guidance and its illumination feature. Subsequent columns illustrate images generated under various specific lighting conditions, with the illumination direction indicated by a sphere.
• Figure 4: Illumination effect control of an invisible light source. Given the same illumination direction prompt, our method is able to generate multiple variants of illumination effect with respect to the strength of the light source.
• Figure 5: Illumination Property-Guided Image Generation (Using EDM karras2022elucidating): Each pair of columns displays the generated image alongside its corresponding illumination feature. Illumination direction indicated by a sphere.