Masked Depth Modeling for Spatial Perception

TL;DR

Masked Depth Modeling reframes depth sensor gaps as informative masks rather than noise to learn metric-scale, pixel-aligned depth from RGB-D inputs. LingBot-Depth pretrains a ViT-Large to learn joint RGB–depth representations and uses a ConvStack decoder, trained on large-scale synthetic and real RGB-D data to produce robust depth completion and strong monocular depth priors. The results show improved depth accuracy, denser coverage, and aligned RGB–depth latent representations, with strong generalization to video depth, 3D tracking, and real-world robotics without task-specific training. The work provides a scalable data-curation approach and a practical pretraining framework that enhances spatial perception for autonomous systems and robotics.

Abstract

Spatial visual perception is a fundamental requirement in physical-world applications like autonomous driving and robotic manipulation, driven by the need to interact with 3D environments. Capturing pixel-aligned metric depth using RGB-D cameras would be the most viable way, yet it usually faces obstacles posed by hardware limitations and challenging imaging conditions, especially in the presence of specular or texture-less surfaces. In this work, we argue that the inaccuracies from depth sensors can be viewed as "masked" signals that inherently reflect underlying geometric ambiguities. Building on this motivation, we present LingBot-Depth, a depth completion model which leverages visual context to refine depth maps through masked depth modeling and incorporates an automated data curation pipeline for scalable training. It is encouraging to see that our model outperforms top-tier RGB-D cameras in terms of both depth precision and pixel coverage. Experimental results on a range of downstream tasks further suggest that LingBot-Depth offers an aligned latent representation across RGB and depth modalities. We release the code, checkpoint, and 3M RGB-depth pairs (including 2M real data and 1M simulated data) to the community of spatial perception.

Figures (15)

• Figure 1: Enhanced sensor depth powered by our proposed MDM pretraining, which leverages naturally missing depth measurements in RGB-D sensors as masks to learn metric-scale, complete, and accurate depth representations. The resulting LingBot-Depth model serves as a powerful spatial perception prior for downstream applications, including 3D point tracking and dexterous grasping.
• Figure 2: Illustration of the proposed Masked Depth Modeling framework. Depth tokens corresponding to missing sensor measurements are masked, and a ViT encoder learns a joint embedding from the contextual tokens (i.e., the RGB frame) and the remaining unmasked depth tokens. In the decoder stage, latent depth tokens are discarded, and a ConvStack decoder reconstructs the full depth map from latent contextual tokens. We put an unmasked depth map in the bottom-right as the reference.
• Figure 3: Multi-query depth-to-RGB attention visualization. For two scenes, (a) an aquarium with densely packed objects and (b) an indoor shelf with heterogeneous materials, we select three depth query patches (Q1--Q3) and visualize their attention over RGB tokens. Each row shows the masked input depth (with query location marked by $\star$), the attention overlay on the RGB image, and the refined depth output. Different queries attend to distinct, spatially corresponding regions, confirming that the joint embedding captures fine-grained cross-modal geometric--appearance associations. The RGB-D camera we used here is Orbbec Gemini-335.
• Figure 4: Our data curation pipelines. Samples from a total of 2.1M real-captured samples plus 1.0M simulated captures are gather in the top row. In the bottom row, we show the RGB-D inputs and the GT depth maps accordingly.
• Figure 5: Mask ratio distributions for our curated synthetic and real-world RGB-D datasets. We compute the ratio of invalid depth pixels as the original mask ratio. Note that the synthetic data was processed by open-source SGM sgm algorithm, it has more missing measurements in the simulated sensor depth than the real captures.