Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation

TL;DR

Hybrid-depth addresses granularity gaps in self-supervised monocular depth estimation by fusing CLIP's global semantics with DINO's local details under a coarse-to-fine, language-guided framework. The method first learns coarse depth via contrastive language priors on multi-grained features, then refines depth with camera pose and pixel-level alignment using learnable depth prompts, functioning as a plug-in encoder for existing MDE pipelines. Empirical results on KITTI show state-of-the-art improvements and improved BEV perception, underscoring the practical impact of integrating vision-language foundation models into geometric perception tasks. The approach demonstrates that depth-aware prompts and cross-modal alignment can harmonize semantic richness with spatial precision for robust 3D understanding.

Abstract

Current self-supervised monocular depth estimation (MDE) approaches encounter performance limitations due to insufficient semantic-spatial knowledge extraction. To address this challenge, we propose Hybrid-depth, a novel framework that systematically integrates foundation models (e.g., CLIP and DINO) to extract visual priors and acquire sufficient contextual information for MDE. Our approach introduces a coarse-to-fine progressive learning framework: 1) Firstly, we aggregate multi-grained features from CLIP (global semantics) and DINO (local spatial details) under contrastive language guidance. A proxy task comparing close-distant image patches is designed to enforce depth-aware feature alignment using text prompts; 2) Next, building on the coarse features, we integrate camera pose information and pixel-wise language alignment to refine depth predictions. This module seamlessly integrates with existing self-supervised MDE pipelines (e.g., Monodepth2, ManyDepth) as a plug-and-play depth encoder, enhancing continuous depth estimation. By aggregating CLIP's semantic context and DINO's spatial details through language guidance, our method effectively addresses feature granularity mismatches. Extensive experiments on the KITTI benchmark demonstrate that our method significantly outperforms SOTA methods across all metrics, which also indeed benefits downstream tasks like BEV perception. Code is available at https://github.com/Zhangwenyao1/Hybrid-depth.

Figures (5)

• Figure 1: CLIP and DINO exhibit complementary strengths: CLIP excels in capturing global semantic context, while DINO specializes in local spatial detail extraction. However, their fusion is hindered by inherent feature-level mismatches. Direct aggregation strategies like channel concatenation (Fig. 1(a)) result in suboptimal depth representations due to misaligned semantic and spatial features. In contrast, our approach (Fig. 1(b)) employs depth-aware language prompts as a granularity calibrator to align cross-level features into a unified depth hierarchy, ensuring semantic coherence and spatial precision.
• Figure 2: The detailed pipeline of our proposed method. We first aggregate different-grained features from CLIP and DINO for coarse depth sensing under contrastive language guidance (Fig. 2(b.1)), incorporating prior geometric knowledge (Fig. 2(a)). These models are then optimized with the help of the auxiliary camera pose from PoseNet like existing methods to learn a fine depth estimation (Fig. 2(b.2)). It also can be extended to improve the capture of continuous depth variations. By equipping our method, existing self-supervised MDE methods (e.g., Monodepth2 monodepthv2, ManyDepth manydepth21, and Mono-VIFI liu2024mono) achieve significant performance improvements (Fig. 2(c)).
• Figure 3: Left: For the coarse depth sensing stage, we first aggregate the CLIP and DINO features and then design two contrastive learning strategies to endow them with coarse depth sensing capabilities by leveraging geometric priors from self-driving scenes. Right: During the fine depth estimation phrase, different from previous methods, Hybrid-depth not only combines camera pose information from PoseNet but also conducts pixel-wise alignment with learnable depth prompts for hybrid-grained features to learn a fine depth estimation ability.
• Figure 4: Patch selecting and feature concatenating.
• Figure 5: Qualitative comparison with Manydepth manydepth21, SQLDepth WangLiangXuJiaoYu2024 and Monodepth2 with Hybrid-depth on the KITTI dataset. Monodepth2 with Hybrid-depth accurately predicts continuous depth in the ground region while preserving sharp object edges.