LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes

TL;DR

LuxRemix addresses the challenge of post-capture lighting control in indoor scenes by integrating a single-image OLAT lighting decomposition with multi-view harmonization and a real-time relightable 3D Gaussian splatting representation. The method enables independent manipulation of individual near-field light sources (on/off, color, intensity) while ensuring consistency across views and enabling real-time interactive relighting from novel viewpoints. It introduces a three-stage pipeline: synthetic dataset generation for per-light decomposition, cross-view harmonization with Plücker ray embeddings, and per-light RGB parameter fitting on a 3D Gaussian splatting backbone, achieving high-fidelity, view-consistent relighting. This work advances photorealistic indoor relighting with fine-grained light control, with potential impact on photography, cinematography, and virtual production, while noting limitations in generalization to dynamic outdoor scenes and absence of distant HDRI editing.

Abstract

We present a novel approach for interactive light editing in indoor scenes from a single multi-view scene capture. Our method leverages a generative image-based light decomposition model that factorizes complex indoor scene illumination into its constituent light sources. This factorization enables independent manipulation of individual light sources, specifically allowing control over their state (on/off), chromaticity, and intensity. We further introduce multi-view lighting harmonization to ensure consistent propagation of the lighting decomposition across all scene views. This is integrated into a relightable 3D Gaussian splatting representation, providing real-time interactive control over the individual light sources. Our results demonstrate highly photorealistic lighting decomposition and relighting outcomes across diverse indoor scenes. We evaluate our method on both synthetic and real-world datasets and provide a quantitative and qualitative comparison to state-of-the-art techniques. For video results and interactive demos, see https://luxremix.github.io.

Figures (14)

• Figure 1: LuxRemix enables interactive light editing of indoor scenes. Our method decomposes complex scene lighting into one-light-at-a-time (OLAT) sources and ambient lighting, which can be remixed for relighting effects. In the top right, we apply our method to single images, where we can change lights and their colors. LuxRemix also enables multi-view-consistent harmonization of the decomposed lighting across multi-view images. By combining these capabilities, we enable real-time relighting of indoor scenes using 3D Gaussian splatting.
• Figure 2: Overview. Given a multi-view capture of an indoor scene, we apply our single-image lighting decomposition model to generate one-light-at-a-time (OLAT) decompositions. We propagate these decompositions across all views using multi-view lighting harmonization with geometric constraints. Repeating this for each light source yields a complete multi-view OLAT dataset. We train a relightable 3D Gaussian splatting representation on these decompositions, enabling real-time interactive control over each light from any viewpoint.
• Figure 3: Synthetic multi-light data example. We generate 10,000 synthetic 3D scenes with procedurally generated light sources. For each scene, we render equirectangular images for multiple lighting conditions, including fully lit, randomly lit, and separate ambient and one-light-at-a-time (OLAT) configurations. This dataset allows us to train high-quality models to decompose scene lighting into individual light sources. Our models train on perspective views sampled from these equirectangular images.
• Figure 4: Single-image Lighting Decomposition. Given the light masks in the top row, our single-image lighting decomposition model realistically decomposes the scene lighting in the input image (on the left) into four individual one-light-at-a-time (OLAT) light sources.
• Figure 5: Multi-view Lighting Harmonization. We use the first input image to decompose the scene lighting into multiple one-light-at-a-time (OLAT) light components and the ambient lighting (highlighted in blue), which includes all unseen light sources and environment lighting. LuxRemix can turn on all three lamps for the top scene and turn off either lamp for the bottom scene. We then propagate these lighting components consistently across all views using the multi-view input images (top row). Sources: Zip-NeRF BarroMVSH2023, VR-NeRF XuALGBKRPKBLZR2023.