DeFM: Learning Foundation Representations from Depth for Robotics

TL;DR

DeFM tackles the lack of large-scale, depth-focused pretrained encoders for robotics by training a self-supervised, depth-only foundation model on 60.4 million depth images using a DINOv2-style objective. A novel log-depth input normalization preserves metric depth across scales, and the model is distilled into compact CNN and ViT variants (3–30M parameters) for resource-constrained robots. Empirical results show state-of-the-art performance on depth-based classification and segmentation and strong sim-to-real transfer across navigation, locomotion, and manipulation tasks, with pretrained weights released for off-the-shelf use. Overall, DeFM demonstrates that depth-specific foundation representations can generalize across diverse embodied tasks without task-specific fine-tuning, enabling more robust and scalable robotic learning pipelines.

Abstract

Depth sensors are widely deployed across robotic platforms, and advances in fast, high-fidelity depth simulation have enabled robotic policies trained on depth observations to achieve robust sim-to-real transfer for a wide range of tasks. Despite this, representation learning for depth modality remains underexplored compared to RGB, where large-scale foundation models now define the state of the art. To address this gap, we present DeFM, a self-supervised foundation model trained entirely on depth images for robotic applications. Using a DINO-style self-distillation objective on a curated dataset of 60M depth images, DeFM learns geometric and semantic representations that generalize to diverse environments, tasks, and sensors. To retain metric awareness across multiple scales, we introduce a novel input normalization strategy. We further distill DeFM into compact models suitable for resource-constrained robotic systems. When evaluated on depth-based classification, segmentation, navigation, locomotion, and manipulation benchmarks, DeFM achieves state-of-the-art performance and demonstrates strong generalization from simulation to real-world environments. We release all our pretrained models, which can be adopted off-the-shelf for depth-based robotic learning without task-specific fine-tuning. Webpage: https://de-fm.github.io/

Figures (15)

• Figure 1: We present DeFM, a foundation model for depth images. Pretrained using DINOv2 style self-distillation method (III) on a curated depth dataset of 60M images (II), DeFM features achieve state-of-the-art results across several classification and semantic segmentation benchmarks (linear probing) (I). Features obtained from DeFM reveal semantic awareness upon performing PCA despite depth lacking texture and color (V). We distill our largest DeFM model into several efficient CNN networks (IV) to be used for various downstream robotic Reinforcement Learning tasks, including navigation, manipulation, and locomotion (VI). In (I), the scales are linearly scaled to the top performer, and zero performance implies it was not tested.
• Figure 2: Overview of the self-supervised pretraining of DeFM.
• Figure 3: Visualization of different depth input normalization methods. Log normalization effectively captures the overall depth range while preserving fine-grained structure in near-field regions (b, d, f). In contrast, standard metric normalization yields weaker contrast and poorly separated gradients (c, e). In our representation, we stack columns (b), (d), and (f) to form a 3-channel normalized depth input, which preserves metric depth while maintaining robustness across diverse domains.
• Figure 4: Overview of distilling the DeFM-L/14 teacher into CNN backbones. A BiFPN module is added on top of the CNN encoder to produce dense spatial features, which are supervised using the teacher’s spatial tokens. The teacher’s class token provides global supervision to the CNN’s pooled feature representation.
• Figure 5: PCA visualization of the patch features obtained from the DeFM-L/14 encoder when processing depth images of various cups captured by different sensors. The first three PCA components are mapped to the RGB color channels for visualization. Notice the feature consistency of the cup handle (visualized in yellow) across all images, demonstrating that DeFM learns a useful prior for a robotic grasping task. The background is removed by thresholding the first PCA component.