ScribbleLight: Single Image Indoor Relighting with Scribbles

TL;DR

This paper introduces ScribbleLight, a generative model that supports local fine-grained control of lighting effects through scribbles that describe changes in lighting and demonstrates its ability to create different lighting effects from sparse scribble annotations.

Abstract

Image-based relighting of indoor rooms creates an immersive virtual understanding of the space, which is useful for interior design, virtual staging, and real estate. Relighting indoor rooms from a single image is especially challenging due to complex illumination interactions between multiple lights and cluttered objects featuring a large variety in geometrical and material complexity. Recently, generative models have been successfully applied to image-based relighting conditioned on a target image or a latent code, albeit without detailed local lighting control. In this paper, we introduce ScribbleLight, a generative model that supports local fine-grained control of lighting effects through scribbles that describe changes in lighting. Our key technical novelty is an Albedo-conditioned Stable Image Diffusion model that preserves the intrinsic color and texture of the original image after relighting and an encoder-decoder-based ControlNet architecture that enables geometry-preserving lighting effects with normal map and scribble annotations. We demonstrate ScribbleLight's ability to create different lighting effects (e.g., turning lights on/off, adding highlights, cast shadows, or indirect lighting from unseen lights) from sparse scribble annotations.

Figures (8)

• Figure 1: ScribbleLight consists of an Albedo-conditioned Stable image Diffusion model (trained in Stage 1), and a ControlNet (trained in Stage 2) that guides the albedo-conditioned diffusion model for relighting through a latent encoding of the scribbles and normals. To regularize the latent encoding, we jointly train a decoder that predicts the target shading (and normals) from the scribbles (and normals).
• Figure 2: Qualitative comparison of relighting quality between LightIt* kocsis2024lightit, RGB$\leftrightarrow$X zeng2024rgbx and ScribbleLight (Ours) with auto-generated scribbles given a target (GT) image.
• Figure 3: Qualitative comparison of relighting quality of LightIt* kocsis2024lightit, RGB$\leftrightarrow$X zeng2024rgbx and ScribbleLight (Ours) with user-provided hand-drawn scribbles.
• Figure 4: Our method consistently follows the scribble, even with different random seeds, allowing users to select their preferred result.
• Figure 5: Demonstration of ScribbleLight's ability to generate different plausible relit images by turning on and off different lights while maintaining the intrinsics of the input photograph.