TranSplat: Surface Embedding-guided 3D Gaussian Splatting for Transparent Object Manipulation

TL;DR

TranSplat tackles the challenge of dense depth completion for transparent objects where conventional depth sensors fail and NeRF-based methods struggle with non-Lambertian surfaces. It introduces a diffusion-based surface-embedding extractor to produce viewpoint- and illumination-robust surface representations, and a joint Gaussian Splatting framework that renders depth using both RGB and SurfEmb cues. The approach yields superior depth completion on synthetic and real datasets and enables reliable grasp planning for transparent objects, with open-source data and models. Overall, TranSplat provides a practical, fast alternative to NeRF-based methods for transparent-object manipulation by decoupling surface representation from raw intensity and leveraging efficient Gaussian rendering.

Abstract

Transparent object manipulation remains a significant challenge in robotics due to the difficulty of acquiring accurate and dense depth measurements. Conventional depth sensors often fail with transparent objects, resulting in incomplete or erroneous depth data. Existing depth completion methods struggle with interframe consistency and incorrectly model transparent objects as Lambertian surfaces, leading to poor depth reconstruction. To address these challenges, we propose TranSplat, a surface embedding-guided 3D Gaussian Splatting method tailored for transparent objects. TranSplat uses a latent diffusion model to generate surface embeddings that provide consistent and continuous representations, making it robust to changes in viewpoint and lighting. By integrating these surface embeddings with input RGB images, TranSplat effectively captures the complexities of transparent surfaces, enhancing the splatting of 3D Gaussians and improving depth completion. Evaluations on synthetic and real-world transparent object benchmarks, as well as robot grasping tasks, show that TranSplat achieves accurate and dense depth completion, demonstrating its effectiveness in practical applications. We open-source synthetic dataset and model: https://github. com/jeongyun0609/TranSplat

Figures (6)

• Figure 1: TranSplat optimizes 3D Gaussian splatting by jointly training with RGB and surface embeddings as inputs. This approach prevents the opacity of transparent objects from collapsing to zero and ensures smooth rendering, leading to accurate depth completion and reliable grasping points.
• Figure 2: Overview of the TranSplat method for manipulating transparent objects. First, data is collected using a robot manipulator. Next, a latent diffusion model is employed to learn surface embeddings. Then, both surface embeddings and RGB images are used for joint Gaussian optimization. Finally, depth is rendered to enable accurate robotic grasping.
• Figure 3: Surface embeddings visualization for unseen transparent objects.
• Figure 4: Depth completion results of TRansPose (Top) and ClearPose (Bottom) synthetic sequences.
• Figure 5: Depth completion results of TRansPose test sequence 7 and 26.