TransparentGS: Fast Inverse Rendering of Transparent Objects with Gaussians

TL;DR

TransparentGS tackles the problem of fast, high-fidelity inverse rendering of transparent objects under complex illumination. It introduces transparent Gaussian primitives to encode geometry and material properties, a deferred refraction pipeline for sharp specular refraction, and GaussProbe light-field probes to capture ambient and nearby indirect light; the depth-based IterQuery mitigates parallax in probe queries. The method unifies mesh and 3D-GS to handle secondary ray effects, and a multi-stage reconstruction with optimized losses yields high-quality results within about one hour, enabling real-time-like novel-view synthesis and complex scene compositing. This approach advances practical rendering of transparent objects in scenes with nearby contents and non-pinhole cameras, with robust performance, color handling, and support for colored transparencies, while outlining limitations and potential directions for future work in caustics and complex light paths.

Abstract

The emergence of neural and Gaussian-based radiance field methods has led to considerable advancements in novel view synthesis and 3D object reconstruction. Nonetheless, specular reflection and refraction continue to pose significant challenges due to the instability and incorrect overfitting of radiance fields to high-frequency light variations. Currently, even 3D Gaussian Splatting (3D-GS), as a powerful and efficient tool, falls short in recovering transparent objects with nearby contents due to the existence of apparent secondary ray effects. To address this issue, we propose TransparentGS, a fast inverse rendering pipeline for transparent objects based on 3D-GS. The main contributions are three-fold. Firstly, an efficient representation of transparent objects, transparent Gaussian primitives, is designed to enable specular refraction through a deferred refraction strategy. Secondly, we leverage Gaussian light field probes (GaussProbe) to encode both ambient light and nearby contents in a unified framework. Thirdly, a depth-based iterative probes query (IterQuery) algorithm is proposed to reduce the parallax errors in our probe-based framework. Experiments demonstrate the speed and accuracy of our approach in recovering transparent objects from complex environments, as well as several applications in computer graphics and vision.

Figures (18)

• Figure 1: The overview of our TransparentGS pipeline. Each 3D scene is firstly separated into transparent objects and opaque environment using SAM2 ravi2024sam guided by GroundingDINO liu2025grounding. For transparent objects, we propose transparent Gaussian primitives, which explicitly encode both geometric and material properties within 3D Gaussians. And the properties are rasterized into maps for subsequent deferred shading. For the opaque environment, we recover it with the original 3D-GS, and bake it into GaussProbe surrounding the transparent object. The GaussProbe are then queried through our IterQuery algorithm to compute reflection and refraction.
• Figure 2: Difference between forward and deferred refraction strategies. The gray arrows denote the normal attributes $\mathbf{n}_i$ of transparent Gaussian primitives (the blue ellipsoids). Deferred refraction integrates the alpha-weighted normal $\mathcal{N}$ and the alpha-weighted hitting point $\mathcal{X}$, and samples the GaussProbe with the single refracted ray $\omega_t$ (the orange arrow).
• Figure 3: Illustration of our baking pipeline for Gaussian light field probes. Given a set of environmental images with the transparent object removed, we can reconstruct the 3D scene using the original 3D-GS kerbl20233d. We voxelize the scene and place virtual cameras around the bounding box of the transparent object. For each virtual camera, we project the Gaussian primitives onto the tangent plane of the unit sphere, generating tangent-plane Gaussians. Finally, an $\alpha$-blending pass bakes the 360° panoramic color and depth maps at each point, which are subsequently stored in the voxels.
• Figure 4: Illustration of our depth-based iterative Gaussian probes query (IterQuery) algorithm. For clarity, we illustrate four selected probes, with positions $\mathbf{p}_1$, $\mathbf{p}_2$, $\mathbf{p}_3$, and $\mathbf{p}_4$, chosen from the eight (orange circles). The black camera with the red arrow represents the queried ray (refracted or reflected ray), expressed as $\mathbf{o} + t \mathbf{d}$. The goal of the algorithm is to determine the correct $\hat{t}$, such that $\mathbf{o} + \hat{t} \mathbf{d}$ corresponds to the first intersection between the queried ray and the scene, as well as the correct directions $\mathbf{d_1}, \mathbf{d_2}, \mathbf{d_3}, \mathbf{d_4}$.
• Figure 5: Illustration of our mesh-GS fusion strategy. GS$\rightarrow$Mesh: We use transparent Gaussian primitives to guide the primary ray sampling of the SDF, efficiently generating an accurate mesh. Mesh$\rightarrow$GS: We employ the mesh as a proxy for fast secondary ray tracing, which are then utilized in the IterQuery.