Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mirror-3DGS: Incorporating Mirror Reflections into 3D Gaussian Splatting

Jiarui Meng, Haijie Li, Yanmin Wu, Qiankun Gao, Shuzhou Yang, Jian Zhang, Siwei Ma

TL;DR

Mirror-3DGS tackles the challenge of accurate mirror reflections in 3D Gaussian Splatting by introducing a per-Gaussian mirror attribute and a plane-based mirror model to generate a virtual viewpoint behind the mirror. It constructs mirrored and original views via a learned mirror plane $\pi=(\boldsymbol{n}_{\pi}^{\top}, d)$ and a mirror transformation $\mathbf{P_m}=\mathbf{T_m}\mathbf{P_o}$, then fuses the two renders with a learned mask to produce coherent reflections without ray tracing. A two-stage training regime learns a rough Gaussian scene and the mirror plane (Stage 1) using mirror masks and depth supervision, followed by depth-aware fusion and refinement with a fixed plane (Stage 2). Across synthetic and real scenes, Mirror-3DGS achieves real-time rendering with mirror-region fidelity comparable to or better than Mirror-NeRF, while outperforming vanilla 3DGS in reflective content and significantly outperforming NeRF-based methods in speed.

Abstract

3D Gaussian Splatting (3DGS) has significantly advanced 3D scene reconstruction and novel view synthesis. However, like Neural Radiance Fields (NeRF), 3DGS struggles with accurately modeling physical reflections, particularly in mirrors, leading to incorrect reconstructions and inconsistent reflective properties. To address this challenge, we introduce Mirror-3DGS, a novel framework designed to accurately handle mirror geometries and reflections, thereby generating realistic mirror reflections. By incorporating mirror attributes into 3DGS and leveraging plane mirror imaging principles, Mirror-3DGS simulates a mirrored viewpoint from behind the mirror, enhancing the realism of scene renderings. Extensive evaluations on both synthetic and real-world scenes demonstrate that our method can render novel views with improved fidelity in real-time, surpassing the state-of-the-art Mirror-NeRF, especially in mirror regions.

Mirror-3DGS: Incorporating Mirror Reflections into 3D Gaussian Splatting

TL;DR

Mirror-3DGS tackles the challenge of accurate mirror reflections in 3D Gaussian Splatting by introducing a per-Gaussian mirror attribute and a plane-based mirror model to generate a virtual viewpoint behind the mirror. It constructs mirrored and original views via a learned mirror plane and a mirror transformation , then fuses the two renders with a learned mask to produce coherent reflections without ray tracing. A two-stage training regime learns a rough Gaussian scene and the mirror plane (Stage 1) using mirror masks and depth supervision, followed by depth-aware fusion and refinement with a fixed plane (Stage 2). Across synthetic and real scenes, Mirror-3DGS achieves real-time rendering with mirror-region fidelity comparable to or better than Mirror-NeRF, while outperforming vanilla 3DGS in reflective content and significantly outperforming NeRF-based methods in speed.

Abstract

3D Gaussian Splatting (3DGS) has significantly advanced 3D scene reconstruction and novel view synthesis. However, like Neural Radiance Fields (NeRF), 3DGS struggles with accurately modeling physical reflections, particularly in mirrors, leading to incorrect reconstructions and inconsistent reflective properties. To address this challenge, we introduce Mirror-3DGS, a novel framework designed to accurately handle mirror geometries and reflections, thereby generating realistic mirror reflections. By incorporating mirror attributes into 3DGS and leveraging plane mirror imaging principles, Mirror-3DGS simulates a mirrored viewpoint from behind the mirror, enhancing the realism of scene renderings. Extensive evaluations on both synthetic and real-world scenes demonstrate that our method can render novel views with improved fidelity in real-time, surpassing the state-of-the-art Mirror-NeRF, especially in mirror regions.
Paper Structure (10 sections, 15 equations, 4 figures, 1 table)

This paper contains 10 sections, 15 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminaries
  4. Mirror-aware 3D Gaussian Representation
  5. Virtual Mirrored Viewpoint Construction and Image Fusion
  6. Two-Stage Training Strategy
  7. Experiments
  8. Experimental Setup
  9. Experimental Results
  10. Conclusion

Figures (4)

  • Figure 1: The synthesized novel views and depth maps. 3DGS confuses mirror reflection, resulting in erroneous mirror depth. Ours properly handles content in the mirror with depth map almost identical to GT.
  • Figure 2: The underlying principles in handling mirrored contents. a) Mirror-NeRF employs a technique involving ray tracing and sampling to achieve mirror reflection. b) 3DGS mistakenly considers objects reflected by the mirror to be placed behind it, resulting the floaters behind the mirror. c) Our proposed Mirror-3DGS, captures the reflected objects by simulating a novel viewpoint situated behind the mirror.
  • Figure 3: The illustration of our two-stage training pipeline. In the Stage 1, we use the mirror mask and view image with mirror content replaced by red to learn the mirror plane and a coarse 3D Gaussian representation of the scene. In the Stage 2,based on the estimated mirror plane, we fuse views from the original and mirrored viewpoints to form the final rendered result, further optimizing the 3D Gaussians.
  • Figure 4: Qualitative comparison of novel view synthesis. Two raws display washroom and discussion room scenes. The top-right shows magnified mirror regions, and the bottom-right shows predicted mirror masks. PSNR is calculated only for the mirror regions. Both Mirror-NeRF and Mirror-3DGS accurately predict mirror masks, with Mirror-3DGS achieving the best performance.