MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections

TL;DR

MirrorGaussian introduces the first real-time, mirror-aware reconstruction framework for scenes containing mirrors by extending 3D Gaussian Splatting with a dual-rendering strategy. It leverages mirror symmetry to render both the real-world Gaussians and their reflected counterparts across an estimated mirror plane, enabling differentiable rasterization and end-to-end optimization. The approach adds per-Gaussian mirror labels and a three-stage training pipeline that refines geometry, the mirror plane, and the mirror mask, achieving high-fidelity reflections and practical scene editing such as adding mirrors or objects. Experiments across four real-world mirror scenes demonstrate state-of-the-art quality with real-time rendering, while also highlighting limitations such as the need for mirror segmentation and a modest speed penalty from dual rendering.

Abstract

3D Gaussian Splatting showcases notable advancements in photo-realistic and real-time novel view synthesis. However, it faces challenges in modeling mirror reflections, which exhibit substantial appearance variations from different viewpoints. To tackle this problem, we present MirrorGaussian, the first method for mirror scene reconstruction with real-time rendering based on 3D Gaussian Splatting. The key insight is grounded on the mirror symmetry between the real-world space and the virtual mirror space. We introduce an intuitive dual-rendering strategy that enables differentiable rasterization of both the real-world 3D Gaussians and the mirrored counterpart obtained by reflecting the former about the mirror plane. All 3D Gaussians are jointly optimized with the mirror plane in an end-to-end framework. MirrorGaussian achieves high-quality and real-time rendering in scenes with mirrors, empowering scene editing like adding new mirrors and objects. Comprehensive experiments on multiple datasets demonstrate that our approach significantly outperforms existing methods, achieving state-of-the-art results. Project page: https://mirror-gaussian.github.io/.

Figures (7)

• Figure 1: The ground truth and renderings of our MirrorGaussian and state-of-the-art methods in the Coffee-House scene from our dataset. Compared to existing methods, our MirrorGaussian achieves both high-quality and real-time rendering (top row), and empowers scene editing such as inserting new objects and mirrors (bottom row).
• Figure 2: Overview of MirrorGaussian. MirrorGaussian is grounded on the mirror symmetry between the real-world scene and its counterpart in the mirror. We first reflect the 3D Gaussians $P_r$ about the mirror plane $\mathcal{P}$ to obtain its mirrored counterpart $P_m$. Then, we rasterize $P_r$ to get the real-world image $I_r$ and the mirror mask $M$, and rasterize $P_m$ to get the mirror image $I_m$. The final image $I$ is composited by $I_r$ and $I_m$ using $M$. $I$ and $M$ are supervised by the captured image $I_{gt}$ and its annotated mirror mask $M_{gt}$, respectively. Note that for the sake of visual simplicity, $P_r$ and $P_m$ have been cropped.
• Figure 3: Reflecting a real-world 3D Gaussian $\mathcal{G}$ with mean $\mu$ and rotation $R$ about the mirror plane (blue line) to get the mirrored 3D Gaussian $\hat{\mathcal{G}}$. (a) illustrates how the mean $\hat{\mu}$ and the rotation $\hat{R}$ of the mirrored 3D Gaussian $\hat{\mathcal{G}}$ are derived using the reflection function $\mathcal{F}$. (b) illustrates how we derive the view-dependent color of $\hat{\mathcal{G}}$. Instead of modifying the SH coefficients according to the reflection, we directly reflect the view direction for $\hat{\mathcal{G}}$ to get the view-dependent color.
• Figure 4: Strategy for yielding the mirror's initial position. The mirror's edges from each frame's mask are extracted using dilation and erosion. Employing SfM, corresponding 2D-3D point pairs are established, allowing reconstruction of 3D points along the mirror edges. Finally, a plane is fitted to the aggregated 3D points from all frames.
• Figure 5: Reconstructed samples with different methods across various scenes. Top row: Meeting Room. Second row: Coffee House. Third row: Lounge. Bottom row: Market.