LightLab: Controlling Light Sources in Images with Diffusion Models

TL;DR

LightLab tackles the challenge of enabling explicit, parametric relighting from a single image by fine-tuning a diffusion model on a compact set of real RAW image pairs augmented with large-scale synthetic renders. It leverages the linearity of light to synthesize paired data and defines a relit sequence $\mathbf{i}_{\text{relit}}(\alpha, \gamma, \mathbf{c}_{\text{t}}; \mathbf{i}_{\text{amb}}, \mathbf{i}_{\text{change}}) = \alpha \mathbf{i}_{\text{amb}} + \gamma \mathbf{i}_{\text{change}} \mathbf{c}$ with $\mathbf{c} = \mathbf{c}_{\text{t}} \odot \mathbf{c}_{\text{o}}^{-1}$ to control ambient, intensity, and color. A diffusion model is conditioned with spatial signals (input image, depth, and light-source masks) and global signals (ambient level and tone-mapping choice) to produce photorealistic relighting with plausible shadows and reflections. Empirical results show that a mixture of real and synthetic data yields the best generalization, and a user study favors LightLab over prior methods for precise, physically plausible lighting edits. The work demonstrates the practicality of physics-informed data generation combined with diffusion-based editing for controllable image illumination, with applications ranging from post-capture relighting to lighting-consistent animation.

Abstract

We present a simple, yet effective diffusion-based method for fine-grained, parametric control over light sources in an image. Existing relighting methods either rely on multiple input views to perform inverse rendering at inference time, or fail to provide explicit control over light changes. Our method fine-tunes a diffusion model on a small set of real raw photograph pairs, supplemented by synthetically rendered images at scale, to elicit its photorealistic prior for relighting. We leverage the linearity of light to synthesize image pairs depicting controlled light changes of either a target light source or ambient illumination. Using this data and an appropriate fine-tuning scheme, we train a model for precise illumination changes with explicit control over light intensity and color. Lastly, we show how our method can achieve compelling light editing results, and outperforms existing methods based on user preference.

Figures (23)

• Figure 1: Post processing pipeline.Top row. From a pair of real (raw) photograph pairs, we first isolate the target light change $\textbf{i}_\text{change}$. Bottom row. For synthetic data, we render each light component separately. After light disentanglement both domains undergo light arithmetic to create parameterized sequences of images $\textbf{i}_\text{relit}\left( \alpha, \gamma, \textbf{c}_\text{t} \right)$, which are later tone mapped to SDR (either together or separately).
• Figure 2: Conditioning Signals. Spatial conditions (input image, target light mask and depth map) are embedded to the latent dimensions and concatenated to the input noise. Light intensity and color control are applied by scaling the intensity and color of the target light mask. Global controls (ambient light intensity and tone-mapping value) are projected to text embedding dimension and inserted through cross-attention.
• Figure 3: Tone mapping strategy. A sequence of images of increasing light intensity, tone mapped either separately or together. Top row. The images tone mapped separately, notice how the light source intensity appears constant when lit, while ambient light appears to be dimmed. Bottom row. Tone mapped together.
• Figure 4: Intensity control. Fine-grained control over a target light's intensity using our method. Values represent the relative intensity change with respect to the source image.
• Figure 5: Color Control. We turn on the street lamp in the input image (top left) with different colors. Top row. artificial light blackbody temperatures. Bottom row arbitrary non-natural RGB colors.