Joint Shadow Generation and Relighting via Light-Geometry Interaction Maps

TL;DR

Light-Geometry Interaction maps are proposed, a novel representation that encodes light-aware occlusion from monocular depth that bridges geometry-inspired rendering with generative modeling, enabling efficient, physically consistent shadow generation and relighting.

Abstract

We propose Light-Geometry Interaction (LGI) maps, a novel representation that encodes light-aware occlusion from monocular depth. Unlike ray tracing, which requires full 3D reconstruction, LGI captures essential light-shadow interactions reliably and accurately, computed from off-the-shelf 2.5D depth map predictions. LGI explicitly ties illumination direction to geometry, providing a physics-inspired prior that constrains generative models. Without such prior, these models often produce floating shadows, inconsistent illumination, and implausible shadow geometry. Building on this representation, we propose a unified pipeline for joint shadow generation and relighting - unlike prior methods that treat them as disjoint tasks - capturing the intrinsic coupling of illumination and shadowing essential for modeling indirect effects. By embedding LGI into a bridge-matching generative backbone, we reduce ambiguity and enforce physically consistent light-shadow reasoning. To enable effective training, we curated the first large-scale benchmark dataset for joint shadow and relighting, covering reflections, transparency, and complex interreflections. Experiments show significant gains in realism and consistency across synthetic and real images. LGI thus bridges geometry-inspired rendering with generative modeling, enabling efficient, physically consistent shadow generation and relighting.

Figures (21)

• Figure 1: Effectiveness of our joint shadow generation and relighting pipeline. Our method produces realistic, texture-aware shadows consistent with object and scene geometry, while preserving faithful relighting across diverse materials such as wood, leather, metal, and glass. (a-c) Multiple object interactions. (d-e) Generalization to multiple light sources.
• Figure 2: Overview of the proposed method. Our approach uses a bridge-matching strategy to transform shadow-free latent codes ($z_0$) into shadowed counterparts ($z_1$), conditioned on global light cues (e.g., light color, radius) and image-derived light–geometry interaction maps. The key novelty lies in generating these interaction maps from image ($x_0$) and light (see \ref{['subsec:occlusion']} for details).
• Figure 3: Elevation difference calculation. Scene objects are shown in blue. Each 2D pixel is lifted to 3D at point ${\bm{p}}$ , from which a ray is cast toward the light source ${\bm{l}}$. Along this ray, we uniformly sample $n$points ${\bm{S}}$ within the valid front-facing camera frustum. Each sampled point is reprojected onto the image plane to retrieve its depth from the predicted depth map, yielding a set of reprojected points ${\bm{S}}'$. The elevation angles ${\bm{e}}^s$ of these reprojected points are then compared with the elevation angle ${\bm{e}}^l$ of the light ray to compute elevation difference ${\bm{e}}^d$. If the light is occluded when viewed from point ${\bm{p}}$, that point is likely to lie in shadow.
• Figure 4: Example of LGI maps. They indicate shadows cast in the environment and also reveal self-shadowing and shading effects, which can be useful for relighting.
• Figure 5: Qualitative comparison with LBM on synthetic object images. Our method produces shadows and relighting effects consistent with geometry and lighting conditions.