PI-Light: Physics-Inspired Diffusion for Full-Image Relighting

TL;DR

π-Light introduces a physics-inspired diffusion framework for full-image relighting that addresses data scarcity, physical plausibility, and generalization through a two-stage pipeline. The first stage performs inverse neural rendering to jointly estimate intrinsic components (albedo, normals, roughness, metallic) with batch-aware cross-batch attention, while the second stage uses a physics-guided forward rendering module to synthesize relit images under controllable lighting, governed by a physics-inspired loss. A new object-and-scene dataset under controlled lighting supports supervised training and benchmarking, enabling strong generalization to real-world scenes with fewer data. Quantitative and qualitative results demonstrate superior performance in both inverse and forward rendering tasks, with improved albedo preservation, lighting control, and material fidelity, suggesting practical impact for AR/VR, film production, and digital content creation. Limitations include latent-RGB misalignment and scope constraints on rear-hemisphere lighting, pointing to future work in richer material models and extended illumination control.

Abstract

Full-image relighting remains a challenging problem due to the difficulty of collecting large-scale structured paired data, the difficulty of maintaining physical plausibility, and the limited generalizability imposed by data-driven priors. Existing attempts to bridge the synthetic-to-real gap for full-scene relighting remain suboptimal. To tackle these challenges, we introduce Physics-Inspired diffusion for full-image reLight ($π$-Light, or PI-Light), a two-stage framework that leverages physics-inspired diffusion models. Our design incorporates (i) batch-aware attention, which improves the consistency of intrinsic predictions across a collection of images, (ii) a physics-guided neural rendering module that enforces physically plausible light transport, (iii) physics-inspired losses that regularize training dynamics toward a physically meaningful landscape, thereby enhancing generalizability to real-world image editing, and (iv) a carefully curated dataset of diverse objects and scenes captured under controlled lighting conditions. Together, these components enable efficient finetuning of pretrained diffusion models while also providing a solid benchmark for downstream evaluation. Experiments demonstrate that $π$-Light synthesizes specular highlights and diffuse reflections across a wide variety of materials, achieving superior generalization to real-world scenes compared with prior approaches.

Figures (28)

• Figure 1: $\pi$-Light enables full-image relighting through physics-inspired diffusion models. It achieves favorable results in both a) inverse rendering and b) neural forward rendering stages, enabling precise control over lighting.
• Figure 2: Comparisons between IC-Light zhangscaling and Ours. The state-of-the-art foreground relighting method still suffers from inconsistencies in maintaining albedo uniformity and lacks physically plausible control over lighting direction.
• Figure 3: Overview of $\pi$-Light.Stage 1: Inverse Neural Rendering. Given a 2D image as input, this stage repurposes a pretrained image diffusion model to simultaneously predict four intrinsic components: albedo, normal, roughness, and metallic. Stage 2: Neural Forward Rendering. Given the input image, the physical intrinsics from Stage 1, and a target lighting condition, this stage tames the image diffusion model to generate the relit image along with its diffuse and specular shading, guided by a physics-inspired light transport prior.
• Figure 4: Diffuse Shading Loss. We incorporate a manually computed loss using the Lambertian model, without requiring ground truth annotations of diffuse shading.
• Figure 5: Qualitative comparisons on our Scene200 test dataset. From top to bottom, the results correspond to albedo, normal, roughness, and metallic comparisons. Our method produces more detailed and accurate results, even in specular reflection regions.