D3RoMa: Disparity Diffusion-based Depth Sensing for Material-Agnostic Robotic Manipulation

TL;DR

D3RoMa introduces a disparity-diffusion framework for stereo depth estimation that explicitly handles challenging materials like transparency and gloss by conditioning a denoising diffusion model on left/right images and raw disparity. The method integrates stereo geometry through classifier-guided reverse sampling, guided by a geometry loss that enforces left-right consistency, and is trained on a new scene-level synthetic dataset (HISS) with realistic IR patterns. Empirical results show state-of-the-art performance on multiple benchmarks, including DREDS, SynTODD, ClearPose, and SceneFlow, and demonstrate clear improvements in robotic manipulation tasks across tabletop and mobile scenarios. The work highlights the importance of diverse training data and geometry-guided diffusion for robust depth perception in real-world robotics, while acknowledging iterative inference as a limitation to be addressed in future work.

Abstract

Depth sensing is an important problem for 3D vision-based robotics. Yet, a real-world active stereo or ToF depth camera often produces noisy and incomplete depth which bottlenecks robot performances. In this work, we propose D3RoMa, a learning-based depth estimation framework on stereo image pairs that predicts clean and accurate depth in diverse indoor scenes, even in the most challenging scenarios with translucent or specular surfaces where classical depth sensing completely fails. Key to our method is that we unify depth estimation and restoration into an image-to-image translation problem by predicting the disparity map with a denoising diffusion probabilistic model. At inference time, we further incorporated a left-right consistency constraint as classifier guidance to the diffusion process. Our framework combines recently advanced learning-based approaches and geometric constraints from traditional stereo vision. For model training, we create a large scene-level synthetic dataset with diverse transparent and specular objects to compensate for existing tabletop datasets. The trained model can be directly applied to real-world in-the-wild scenes and achieve state-of-the-art performance in multiple public depth estimation benchmarks. Further experiments in real environments show that accurate depth prediction significantly improves robotic manipulation in various scenarios.

Figures (14)

• Figure 1: Generalizability of D$^3$RoMa in the real world. Our method robustly predicts transparent (bottles) and specular (basin and cups) object depths in tabletop environments and beyond. RGB image, pseudo colorized raw disparity map, our prediction, and point cloud are displayed for each case of a total of 6 frames captured by camera RealSense D415 and D435. * RGB and depth images are not aligned for the D435 camera for better visualization.
• Figure 2: Disparity diffusion with stereo-geometry guidance. Our disparity diffusion-based depth sensing framework takes the raw disparity map $\tilde{D}$ and the left-right stereo image pair $I_l, I_r$ as input. With the geometry prior from the stereo matching between $I_l$ and $I_r$ as guidance for the reverse sampling, our diffusion model can gradually perform the denoising process conditioned on $\tilde{D}$ to predict the restored disparity map $x_0$.
• Figure 3: Qualitative depth completion results on ClearPose. From left to right, there are RGB image, raw depth, ground truth depth rendered using object CAD models, completed depth by TransCG, ImplicitDepth, and our method D$^3$RoMa.
• Figure 4: Qualitative Comparisons with SOTA methods in the Real World. Each row (from left to right) shows the RGB image and disparity results of our method, pre-trained Raft Stereo, Raft Stereo fine-tuned on our dataset, and ASGrasp.
• Figure 5: Robotic Manipulation. We examine our approach on challenging robot manipulation tasks and our performance significantly outperforms all baselines.