TR-Gaussians: High-fidelity Real-time Rendering of Planar Transmission and Reflection with 3D Gaussian Splatting

TL;DR

TR-Gaussians address the challenge of rendering indoor scenes with planar transmission and reflection by coupling 3D Gaussians for the scene with a learnable reflection plane and mirrored Gaussians for reflections. A Fresnel-based, view-dependent weighting blends transmission and reflection, while a multi-stage optimization plus depth-variance, gradient-conflict, and mask losses enforce robust decomposition. The approach demonstrates real-time, high-fidelity novel-view synthesis on datasets with glass panes and mirrors, outperforming state-of-the-art baselines in both quantitative metrics (PSNR/SSIM/LPIPS) and qualitative rendering of reflections. This work enables photorealistic rendering of complex glass phenomena in practical indoor scenes and offers a scalable path to extend 3D Gaussian-based representations to transmission-reflection combined scenes.

Abstract

We propose Transmission-Reflection Gaussians (TR-Gaussians), a novel 3D-Gaussian-based representation for high-fidelity rendering of planar transmission and reflection, which are ubiquitous in indoor scenes. Our method combines 3D Gaussians with learnable reflection planes that explicitly model the glass planes with view-dependent reflectance strengths. Real scenes and transmission components are modeled by 3D Gaussians and the reflection components are modeled by the mirrored Gaussians with respect to the reflection plane. The transmission and reflection components are blended according to a Fresnel-based, view-dependent weighting scheme, allowing for faithful synthesis of complex appearance effects under varying viewpoints. To effectively optimize TR-Gaussians, we develop a multi-stage optimization framework incorporating color and geometry constraints and an opacity perturbation mechanism. Experiments on different datasets demonstrate that TR-Gaussians achieve real-time, high-fidelity novel view synthesis in scenes with planar transmission and reflection, and outperform state-of-the-art approaches both quantitatively and qualitatively.

Figures (12)

• Figure 1: Our rendering pipeline. The pipeline consists of two rendering passes. In the first pass, we render the transmission image $\mathbf{C}_t$ using primary Gaussians, and generate the reflection mask $\mathbf{M}$ by rendering both primary and glass Gaussians. We then compute the raw reflection strength map $\mathbf{R}_{raw}$ using Fresnel reflectance model, and multiply it with the reflection mask $\mathbf{M}$ to obtain the reflection strength map $\mathbf{R}$. In the second pass, we mirror the primary Gaussians across the reflection plane $\mathbf{P}$ to produce the mirrored Gaussians, which are rasterized to generate the raw reflection image $\mathbf{C}_{raw}$ and the reflection intensity map $\mathbf{A}$. These two are multiplied to produce the reflection image $\mathbf{C}_r$. Finally, we blend the transmission image $\mathbf{C}_t$ and the reflection image $\mathbf{C}_r$ using the reflection strength map $\mathbf{R}$ to produce the final image.
• Figure 2: Rendering comparison of 3DGS (left column) and our method (right column) from two viewpoints — in front of the glass (top row) and behind the glass (bottom row). Our method effectively disentangles reflection and transmission components: clear reflections occur solely on the glass’s front side; no reflections are visible from behind. By comparison, 3DGS fails to accurately model reflections on transparent glass, producing many floaters of "reflection ghost" far from the real surface.
• Figure 3: Qualitative comparison with EnvGS xie2024envgs, Spec-Gaussian spec-gaussian, 3DGS 3dgs, MS-NeRF ms-nerf and NeRFReN nerfren on the RTR dataset. We show images rendered from novel test viewpoints. Compared to these baselines, our method produces more accurate and detailed reflections, and yields results closest to the ground truth.
• Figure 4: Qualitative comparison on Mirror-NeRF dataset. Regions with noticeable differences are cropped and shown side-by-side for clear comparison.
• Figure 5: Decomposition results on RTR dataset. From left to right, there are transmission image $\mathbf{C}_r$, reflection image $\mathbf{C}_r$, reflection strength map $\mathbf{R}$, weighted reflection image $\mathbf{R}\cdot\mathbf{C_r}$ and final image $\mathbf{C}$. Our method can achieve natural transmission-reflection decomposition and produce high-quality novel view synthesis results.