LightHarmony3D: Harmonizing Illumination and Shadows for Object Insertion in 3D Gaussian Splatting

Abstract

3D Gaussian Splatting (3DGS) enables high-fidelity reconstruction of scene geometry and appearance. Building on this capability, inserting external mesh objects into reconstructed 3DGS scenes enables interactive editing and content augmentation for immersive applications such as AR/VR, virtual staging, and digital content creation. However, achieving physically consistent lighting and shadows for mesh insertion remains challenging, as it requires accurate scene illumination estimation and multi-view consistent rendering. To address this challenge, we present LightHarmony3D, a novel framework for illumination-consistent mesh insertion in 3DGS scenes. Central to our approach is our proposed generative module that predicts a full 360° HDR environment map at the insertion location via a single forward pass. By leveraging generative priors instead of iterative optimization, our method efficiently captures dominant scene illumination and enables physically grounded shading and shadows for inserted meshes while maintaining multi-view coherence. Furthermore, we introduce the first dedicated benchmark for mesh insertion in 3DGS, providing a standardized evaluation framework for assessing lighting consistency and photorealism. Extensive experiments across multiple real-world reconstruction datasets demonstrate that LightHarmony3D achieves state-of-the-art realism and multi-view consistency.

Figures (4)

• Figure 1: Overview of LightHarmony3D. Given multi-view input images, we reconstruct a hybrid 3DGS–mesh scene and render a base 360$^\circ$ panorama. To capture dominant light emitters, a fine-tuned diffusion model predicts bracketed underexposures, which are fused to reconstruct a HDR environment map via radiometric truncation. To illuminate the enclosed scene, we employ a ray-decoupled visibility formulation that renders the proxy mesh transparent to camera rays while preserving its physical shadow-catching properties. Finally, we perform physically based rendering of the inserted mesh and linearly composite the resulting physically grounded shading and cast shadows over the original 3DGS background to produce seamless, illumination-consistent results.
• Figure 2: Qualitative results on our LH3D synthetic dataset and real scenes from Mip-NeRF360. The inserted object is highlighted with a bounding box. As Mip-NeRF360 provides no insertion ground truth, we additionally show the original background for reference.
• Figure 3: Visual results of the ablation study on LH3D-Ku. Details are zoomed in within the bounding boxes.
• Figure 4: Extensibility. Top: frames from two animation sequences (A1, A2) showing stable illumination across time. Bottom: multi-view insertion results demonstrating consistent shading and shadows from different viewpoints.