Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gaussian Splatting in Mirrors: Reflection-Aware Rendering via Virtual Camera Optimization

Zihan Wang, Shuzhe Wang, Matias Turkulainen, Junyuan Fang, Juho Kannala

TL;DR

This paper tackles the challenge of rendering reflections in 3D Gaussian Splatting by modeling mirror reflections with virtual cameras placed symmetrically around a predicted mirror plane. It introduces depth- and normal-guided plane estimation, a virtual-camera rendering framework, and a photometric-based optimization to refine both the mirror plane and the virtual camera pose, all within a progressive training workflow. The approach yields high-quality, multi-view-consistent reflections, achieving or surpassing state-of-the-art methods like Mirror-NeRF while offering substantially faster rendering. A new real-world mirror dataset supports robust evaluation across varied mirror sizes and shapes, underscoring the method’s practical impact for real-time reflection-aware view synthesis in indoor scenes.

Abstract

Recent advancements in 3D Gaussian Splatting (3D-GS) have revolutionized novel view synthesis, facilitating real-time, high-quality image rendering. However, in scenarios involving reflective surfaces, particularly mirrors, 3D-GS often misinterprets reflections as virtual spaces, resulting in blurred and inconsistent multi-view rendering within mirrors. Our paper presents a novel method aimed at obtaining high-quality multi-view consistent reflection rendering by modelling reflections as physically-based virtual cameras. We estimate mirror planes with depth and normal estimates from 3D-GS and define virtual cameras that are placed symmetrically about the mirror plane. These virtual cameras are then used to explain mirror reflections in the scene. To address imperfections in mirror plane estimates, we propose a straightforward yet effective virtual camera optimization method to enhance reflection quality. We collect a new mirror dataset including three real-world scenarios for more diverse evaluation. Experimental validation on both Mirror-Nerf and our real-world dataset demonstrate the efficacy of our approach. We achieve comparable or superior results while significantly reducing training time compared to previous state-of-the-art.

Gaussian Splatting in Mirrors: Reflection-Aware Rendering via Virtual Camera Optimization

TL;DR

This paper tackles the challenge of rendering reflections in 3D Gaussian Splatting by modeling mirror reflections with virtual cameras placed symmetrically around a predicted mirror plane. It introduces depth- and normal-guided plane estimation, a virtual-camera rendering framework, and a photometric-based optimization to refine both the mirror plane and the virtual camera pose, all within a progressive training workflow. The approach yields high-quality, multi-view-consistent reflections, achieving or surpassing state-of-the-art methods like Mirror-NeRF while offering substantially faster rendering. A new real-world mirror dataset supports robust evaluation across varied mirror sizes and shapes, underscoring the method’s practical impact for real-time reflection-aware view synthesis in indoor scenes.

Abstract

Recent advancements in 3D Gaussian Splatting (3D-GS) have revolutionized novel view synthesis, facilitating real-time, high-quality image rendering. However, in scenarios involving reflective surfaces, particularly mirrors, 3D-GS often misinterprets reflections as virtual spaces, resulting in blurred and inconsistent multi-view rendering within mirrors. Our paper presents a novel method aimed at obtaining high-quality multi-view consistent reflection rendering by modelling reflections as physically-based virtual cameras. We estimate mirror planes with depth and normal estimates from 3D-GS and define virtual cameras that are placed symmetrically about the mirror plane. These virtual cameras are then used to explain mirror reflections in the scene. To address imperfections in mirror plane estimates, we propose a straightforward yet effective virtual camera optimization method to enhance reflection quality. We collect a new mirror dataset including three real-world scenarios for more diverse evaluation. Experimental validation on both Mirror-Nerf and our real-world dataset demonstrate the efficacy of our approach. We achieve comparable or superior results while significantly reducing training time compared to previous state-of-the-art.
Paper Structure (14 sections, 13 equations, 3 figures, 3 tables)

This paper contains 14 sections, 13 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. 3D Gaussian Splatting
  4. Virtual Camera Definition
  5. Mirror Plane Prediction
  6. Virtual Camera Optimization
  7. Progressive Training
  8. Experiments
  9. Training Details
  10. Experimental Results
  11. Mirror-NeRF Dataset Results
  12. Our Real-World Dataset Results
  13. Ablation Study
  14. Conclusion

Figures (3)

  • Figure 1: Pipeline Overview: We extend 3D-GS with depth and normal supervision to initialize a mirror plane in Sec. \ref{['sec:Mirror Plane Prediction']}. Blue region in $\hat{C}$ indicates the mirror. We render both real and virtual camera viewpoints and combine them into a single image in Sec \ref{['sec:Virtual Camera Rendering']}. We further refine virtual camera positions during optimization to achieve photo-realistic mirror reflections in Sec. \ref{['sec:Virtual Camera Optimization']}.
  • Figure 2: The process of virtual camera rendering in Sec. \ref{['sec:Virtual Camera Rendering']}.
  • Figure 3: Qualitative comparisons of ground truth, 3D-GS, and our method on the Living Room, Office, Lounge, Corridor, and Recovery Room scenes from Mirror-NeRF dataset. The smaller image in the upper right corner of each main images is an enlargement of a mirror region.