Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation

TL;DR

This work addresses the difficulty of depth perception for transparent/reflective objects by leveraging the physics-aware priors embedded in large video diffusion models. It introduces TransPhy3D, a synthetic video dataset of 11k scenes with rich transparent materials, and DKT, a video-depth estimator finetuned from a diffusion model via LoRA with a co-training strategy on image and video data. DKT achieves zero-shot state-of-the-art performance on real and synthetic benchmarks (ClearPose, DREDS, TransPhy3D-Test) for depth and normals, and significantly improves temporal consistency, with a compact 1.3B variant running ~0.17 s per frame. The method demonstrates practical impact by boosting grasp success on translucent/reflective surfaces, illustrating how generative priors can enable robust, label-free perception for challenging real-world manipulation. $Diffusion\ knowledge$ of transparency thus enables robust, temporally coherent perception in difficult materials.

Abstract

Transparent objects remain notoriously hard for perception systems: refraction, reflection and transmission break the assumptions behind stereo, ToF and purely discriminative monocular depth, causing holes and temporally unstable estimates. Our key observation is that modern video diffusion models already synthesize convincing transparent phenomena, suggesting they have internalized the optical rules. We build TransPhy3D, a synthetic video corpus of transparent/reflective scenes: 11k sequences rendered with Blender/Cycles. Scenes are assembled from a curated bank of category-rich static assets and shape-rich procedural assets paired with glass/plastic/metal materials. We render RGB + depth + normals with physically based ray tracing and OptiX denoising. Starting from a large video diffusion model, we learn a video-to-video translator for depth (and normals) via lightweight LoRA adapters. During training we concatenate RGB and (noisy) depth latents in the DiT backbone and co-train on TransPhy3D and existing frame-wise synthetic datasets, yielding temporally consistent predictions for arbitrary-length input videos. The resulting model, DKT, achieves zero-shot SOTA on real and synthetic video benchmarks involving transparency: ClearPose, DREDS (CatKnown/CatNovel), and TransPhy3D-Test. It improves accuracy and temporal consistency over strong image/video baselines, and a normal variant sets the best video normal estimation results on ClearPose. A compact 1.3B version runs at ~0.17 s/frame. Integrated into a grasping stack, DKT's depth boosts success rates across translucent, reflective and diffuse surfaces, outperforming prior estimators. Together, these results support a broader claim: "Diffusion knows transparency." Generative video priors can be repurposed, efficiently and label-free, into robust, temporally coherent perception for challenging real-world manipulation.

Figures (9)

• Figure 1: In-the-Wild Qualitative Results. The first and third rows present frames extracted from the input videos, while the second and fourth rows display the predictions. Our method achieves robust depth estimation for transparent objects in arbitrary-length, in-the-wild videos. For the full video, please refer to the Appendix video.
• Figure 2: We present DKT, a foundational model for fine-grained, temporally consistent depth estimation of in-the-wild videos featuring transparent objects of arbitrary lengths.
• Figure 3: Overaview of DKT. DKT starts with a pretrained video diffusion model wan2025 and is finetuned for video depth estimation by concatenating an extra RGB latent with the input latent using LoRA training strategy.
• Figure 4: Rendering Pipeline. Scenarios are constructed using static and parametric 3D assets. RGB, depth, and normal videos are rendered by sampling a circular trajectory within the scene.
• Figure 5: Qualitative comparison on the ClearPose chen2022clearpose. For better visualizing the temporal quality, we show the temporal profiles of each result in green boxes, by slicing the depth values along the time axis at the red line positions.