On the Role of Depth in Surgical Vision Foundation Models: An Empirical Study of RGB-D Pre-training

TL;DR

This work examines whether depth information enhances vision foundation models for surgery through RGB-D pre-training. Using 1.4 million DV frames with pseudo-depth maps, it compares eight ViT-based models across eight surgical datasets spanning object detection, segmentation, depth estimation, and pose estimation, under frozen-backbone and end-to-end fine-tuning. The study finds that explicit geometric pre-training with depth (e.g., MultiMAE, Mask3D) yields substantial improvements, especially for dense tasks, and enables data-efficient learning where 25% of labeled data can surpass full-data RGB-only models; importantly, these gains occur without any inference-time architectural changes. Depth-informed pre-training also mitigates domain shift and yields strong results even when depth maps are imperfect, supporting the practicality of deploying geometry-aware surgical VFMs in real-world settings.

Abstract

Vision foundation models (VFMs) have emerged as powerful tools for surgical scene understanding. However, current approaches predominantly rely on unimodal RGB pre-training, overlooking the complex 3D geometry inherent to surgical environments. Although several architectures support multimodal or geometry-aware inputs in general computer vision, the benefits of incorporating depth information in surgical settings remain underexplored. We conduct a large-scale empirical study comparing eight ViT-based VFMs that differ in pre-training domain, learning objective, and input modality (RGB vs. RGB-D). For pre-training, we use a curated dataset of 1.4 million robotic surgical images paired with depth maps generated from an off-the-shelf network. We evaluate these models under both frozen-backbone and end-to-end fine-tuning protocols across eight surgical datasets spanning object detection, segmentation, depth estimation, and pose estimation. Our experiments yield several consistent findings. Models incorporating explicit geometric tokenization, such as MultiMAE, substantially outperform unimodal baselines across all tasks. Notably, geometric-aware pre-training enables remarkable data efficiency: models fine-tuned on just 25% of labeled data consistently surpass RGB-only models trained on the full dataset. Importantly, these gains require no architectural or runtime changes at inference; depth is used only during pre-training, making adoption straightforward. These findings suggest that multimodal pre-training offers a viable path towards building more capable surgical vision systems.

Figures (15)

• Figure 1: An overview of our contribution. (a) Previous work in surgical VFMs typically use unimodal vision-only pre-training schemes (e.g., DINO, MAE), which limit the geometric/3D understanding of the models. In contrast, we use models which explicitly incorporate geometry into the pre-text task (e.g., Mask3D, MultiMAE), thus improving spatial and semantic understanding of surgical scenes. (b) We fine-tune these models to demonstrate the superiority of multimodal over unimodal pre-training across various geometric, semantic, and dense pixel downstream tasks. All values are normalized; please see Table \ref{['tab:e2e_results']} for numerical results.
• Figure 2: Samples from the pre-training dataset. We curate an in-house RGB-D dataset from videos collected via the da Vinci (DV) system and generate pseudo-labeled depth maps via FoundationStereo wen2025foundationstereo, which is robust against corruptions such as blur and smoke.
• Figure 3: Comparison of downstream task performance on eight surgical datasets. Solid inner bars represent performance with a frozen backbone (linear probing), while hatched outer bars represent end-to-end fine-tuning. Vertical dotted lines distinguish between unimodal and multimodal models. Exact numerical results are provided in Appendix \ref{['app:quant_results']}.
• Figure 4: Data efficiency analysis comparing model performance at 25%, 50%, 75%, and 100% training data availability. Solid lines indicate surgical domain (DV) pre-training and dashed lines indicate ImageNet pre-training. The horizontal dotted line marks the baseline performance of MAE (DV) trained on 100% of the data. Numerical results are provided in Appendix \ref{['app:quant_results']}.
• Figure 5: Demonstration of a naive multimodal DINOv2 training scheme. During training, both the teacher and student can receive crops of either modality, thus inducing learning of a joint probability distribution.