D3DR: Lighting-Aware Object Insertion in Gaussian Splatting

TL;DR

D3DR addresses the challenge of natural-looking object insertion into 3D Gaussian Splatting scenes by leveraging diffusion models to enforce lighting consistency. The method combines diffusion model personalization (DreamBooth with IC-Light augmentation) to tailor the object to varied illuminations, followed by a Delta Denoising Score–based optimization to relight and refine the object's appearance within the scene. Key contributions include a two-stage pipeline (diffusion personalization and 2-step DDS), a depth-conditioned ControlNet for geometric consistency, and comprehensive ablations and comparisons showing improvements in relighting quality (e.g., PSNR up to 0.5 and SSIM up to 0.15). The approach enables more realistic 3D object insertion without environment maps or full physically-based rendering, with practical implications for neural graphics applications and extended 3D content editing.

Abstract

Gaussian Splatting has become a popular technique for various 3D Computer Vision tasks, including novel view synthesis, scene reconstruction, and dynamic scene rendering. However, the challenge of natural-looking object insertion, where the object's appearance seamlessly matches the scene, remains unsolved. In this work, we propose a method, dubbed D3DR, for inserting a 3DGS-parametrized object into 3DGS scenes while correcting its lighting, shadows, and other visual artifacts to ensure consistency, a problem that has not been successfully addressed before. We leverage advances in diffusion models, which, trained on real-world data, implicitly understand correct scene lighting. After inserting the object, we optimize a diffusion-based Delta Denoising Score (DDS)-inspired objective to adjust its 3D Gaussian parameters for proper lighting correction. Utilizing diffusion model personalization techniques to improve optimization quality, our approach ensures seamless object insertion and natural appearance. Finally, we demonstrate the method's effectiveness by comparing it to existing approaches, achieving 0.5 PSNR and 0.15 SSIM improvements in relighting quality.

Figures (41)

• Figure 1: Overview of the task. Our method aims to insert an object into a designated location in a scene, both represented in 3DGS parametrization, followed by adjusting the object's appearance to match the scene's lighting. The final result is a 3DGS scene that includes both the input scene and the object with corrected lighting.
• Figure 2: Pipeline overview. The method is able to perform 3D Object insertion of a 3DGS object into a 3DGS scene with object light correction. The whole pipeline consists of two steps. 1) a diffusion model is personalized on the object, using framework proposed by DreamBooth dreambooth and < ktn> as a rare token. 2) 2-step-DDS is utilized to adjust the object appearance after 3DGS insertion. Fire means that the parameters are optimized (for UNet during personalization and object parameters during 2-step-DDS).
• Figure 3: DDS image editing and lighting dependence. The first column shows the initial images of a cup: the top image has correct lighting, while the bottom one has incorrect lighting. The second column presents the edited outputs using the classical DDS loss. The initial prompt $y_{init}$ is a cup on a plate, and the target prompt $y_{tgt}$ is a statue head on a plate. The results demonstrate that DDS inherits object lighting: the statue head in the top row has correct lighting, while the one in the bottom row retains the incorrect lighting of the cup.
• Figure 4: DDS refines object appearance after insertion. The first row illustrates the optimization process for a cup inserted into an image. The rightmost image represents the ground-truth cup on a plate. The second image shows an inserted cup with incorrect lighting, despite identical global lighting conditions. Images 3–7 depict the cup’s gradual adaptation through DDS optimization. The second row presents a similar experiment under different lighting conditions. The final results closely match the appearance of the ground-truth cups.
• Figure 5: Comparison with other methods. The first column shows ground truth object insertions, the second column presents results from our method, the third column shows TIP-Editor results, and the fourth column displays R3DG results. The rows represent different scenes, such as kitchen_1, kitchen_2, and bathroom_2.