GR3EN: Generative Relighting for 3D Environments

TL;DR

GR3EN introduces a novel pipeline for relighting full 3D scene reconstructions by distilling relit video sequences back into a 3D representation. It sidesteps the ill-posed inverse rendering problem by leveraging a fine-tuned video diffusion model to relight rendering outputs conditioned on explicit target lighting, and then distills that information into Zip-NeRF-based 3D representations. The approach supports fine-grained per-light control and can render novel views under new lighting, including real-world scenes. Experiments on synthetic and real data show photorealistic global illumination effects and robust generalization, with ablations supporting design choices. The method enables intuitive lighting edits for 3D reconstructions, with potential for enhanced content creation and visualization.

Abstract

We present a method for relighting 3D reconstructions of large room-scale environments. Existing solutions for 3D scene relighting often require solving under-determined or ill-conditioned inverse rendering problems, and are as such unable to produce high-quality results on complex real-world scenes. Though recent progress in using generative image and video diffusion models for relighting has been promising, these techniques are either limited to 2D image and video relighting or 3D relighting of individual objects. Our approach enables controllable 3D relighting of room-scale scenes by distilling the outputs of a video-to-video relighting diffusion model into a 3D reconstruction. This side-steps the need to solve a difficult inverse rendering problem, and results in a flexible system that can relight 3D reconstructions of complex real-world scenes. We validate our approach on both synthetic and real-world datasets to show that it can faithfully render novel views of scenes under new lighting conditions.

Figures (18)

• Figure 1: Our method provides fine-grained control over the lighting of a 3D scene. Given (a) multi-view input images, we first (b) reconstruct a 3D representation. Given target lighting conditions (here turning off the left overhead light and turning on the lamp on the dresser), our method outputs a (c) relit 3D scene, which can be used to render (d) novel views of the scene under the target lighting.
• Figure 2: Our pipeline for 3D scene relighting. Starting with a 3D reconstruction (top right), we first render a video of the scene along a chosen camera path (top left). We then use our relighting diffusion model to relight the input video given a target lighting, which is provided as a video whose values specify the color and intensity of the target light sources. Finally, the output relit video is distilled back into the 3D representation (bottom right), resulting in a relit 3D scene that can be rendered from novel viewpoints.
• Figure 3: A visualization of our "one-light-at-a-time" data synthesis pipeline. We (a) separately render the scene illuminated by each of the light sources, (b) multiply each one by its target light color, and (c) combine all light sources according to Equation \ref{['eq:olat_combo']} to get the target illumination.
• Figure 4: Real-world relighting results on the Eyeful Tower dataset. The top row shows a frame from the input video (and an inset on the right), with the target lighting condition displayed in the bottom-right corner of the input frame. The target light requires turning off the right half of the overhead lights as well as the light coming from outside the room. The remaining rows show the relit results obtained by Nano Banana, LightLab, and our method. Note that our method produces realistic edits with convincing global illumination effects like the shadow cast by the object on the table. Please refer to the supplementary material and our webpage for video results.
• Figure 5: 3D light insertion. We add a new out-of-distribution light source to a real reconstructed 3D scene. Despite the training data not containing this type of light, and despite the relighting model being trained only using synthetic data, our method still obtains realistic results. Note in particular the consistent global illumination effects observed for example in the shadows of the objects on the table.