Towards Depth Foundation Model: Recent Trends in Vision-Based Depth Estimation

TL;DR

Vision-based depth estimation lags behind sensor-based sensing in generalization and stability, prompting a shift toward depth foundation models. This survey maps architectural and learning paradigm progress across monocular, stereo, multi-view, and monocular video depth estimation, emphasizing large-scale data, Transformer- and diffusion-based priors, and diverse datasets. It catalogs datasets, key methods, and training strategies, while identifying core challenges in data quality, cross-domain generalization, and cross-task integration. The work frames a roadmap for robust, scalable depth perception systems with broad implications for 3D reconstruction, view synthesis, video world modeling, and robotics.

Abstract

Depth estimation is a fundamental task in 3D computer vision, crucial for applications such as 3D reconstruction, free-viewpoint rendering, robotics, autonomous driving, and AR/VR technologies. Traditional methods relying on hardware sensors like LiDAR are often limited by high costs, low resolution, and environmental sensitivity, limiting their applicability in real-world scenarios. Recent advances in vision-based methods offer a promising alternative, yet they face challenges in generalization and stability due to either the low-capacity model architectures or the reliance on domain-specific and small-scale datasets. The emergence of scaling laws and foundation models in other domains has inspired the development of "depth foundation models": deep neural networks trained on large datasets with strong zero-shot generalization capabilities. This paper surveys the evolution of deep learning architectures and paradigms for depth estimation across the monocular, stereo, multi-view, and monocular video settings. We explore the potential of these models to address existing challenges and provide a comprehensive overview of large-scale datasets that can facilitate their development. By identifying key architectures and training strategies, we aim to highlight the path towards robust depth foundation models, offering insights into their future research and applications.

Figures (10)

• Figure 1: Scaling trends in model capacity and data volume for depth estimation. Each point represents a published method, positioned by its approximate model size (bottom axis, logarithmic scale of parameter count) and dataset size (left axis, logarithmic scale of training images), and colored by publication year. Early models (lightest shading, before 2020) relied on sub-million-parameter networks trained on only thousands of images, yielding limited generalization. Over 2020–2021 (light blue shading), methods grew to tens of millions of parameters and hundreds of thousands of image datasets. In 2022-2023 (gray shading), methods continued to increase in size and scalable training, with around one billion parameters and datasets ranging from several million to ten million images. The most recent models (darkest shading, 2024–2025) further scale up to the billions of parameters and utilize over ten million images, with strong generalization ability, and demonstrate the potential evolution toward depth foundation models.
• Figure 2: Overview of the key-idea paradigm evolution of monocular image depth estimation. From early direct regression and classification methods, through affine-invariant and canonical camera depth estimation, monocular depth estimation models have shown increasingly stronger generalization capabilities, paving the way for the emergence of depth foundation models.
• Figure 3: Representative pipelines of monocular image depth estimation. The pink component denotes operations without learnable parameters or with fixed parameters, while the blue component indicates operations with optimizable parameters. Vision Transformer-based approaches, leveraging their lightweight architectures, enable real-time monocular depth estimation. However, due to the presence of convolutional operations in their architectures, they may lose detailed features. Diffusion Model-based methods treat RGB images as conditional inputs, effectively preserving fine-grained details. Nevertheless, their denoising processes impose computational costs, making it challenging to achieve real-time performance.
• Figure 4: Overview of the key-idea paradigm evolution of stereo image depth estimation. The paradigms have transitioned from cost-volume methods to attention mechanisms and iterative optimization techniques to effectively match the features of stereo images. The incorporation of monocular and diffusion priors facilitates large-scale training, paving the way for foundation models.
• Figure 5: Representative pipelines of stereo image depth estimation. The first architecture is for scalable training, which leverages all available datasets along with pseudo stereo pairs synthesized from a monocular dataset, to train a foundation model. The second architecture is migrating knowledge from a monocular foundation model to the stereo model, making it possible to achieve a stereo foundation model from relatively small-scale training datasets.