Bridging Geometric and Semantic Foundation Models for Generalized Monocular Depth Estimation

TL;DR

BriGeS tackles monocular depth estimation by bridging geometric cues from a depth foundation model with semantic cues from a segmentation foundation model. It introduces the Bridging Gate, a cross-attention followed by self-attention module, and an Attention Temperature Scaling technique to avoid over-concentration in attention maps, enabling robust fusion while freezing the backbone encoders. Training focuses on the Bridging Gate with affine-invariant losses and selective gradient terms, achieving strong generalization across unseen datasets and complex scenes with reduced training resources. The approach yields significant improvements over state-of-the-art methods on challenging benchmarks, and highlights a practical path toward more memory-efficient semantic-aware depth representations through future distillation.

Abstract

We present Bridging Geometric and Semantic (BriGeS), an effective method that fuses geometric and semantic information within foundation models to enhance Monocular Depth Estimation (MDE). Central to BriGeS is the Bridging Gate, which integrates the complementary strengths of depth and segmentation foundation models. This integration is further refined by our Attention Temperature Scaling technique. It finely adjusts the focus of the attention mechanisms to prevent over-concentration on specific features, thus ensuring balanced performance across diverse inputs. BriGeS capitalizes on pre-trained foundation models and adopts a strategy that focuses on training only the Bridging Gate. This method significantly reduces resource demands and training time while maintaining the model's ability to generalize effectively. Extensive experiments across multiple challenging datasets demonstrate that BriGeS outperforms state-of-the-art methods in MDE for complex scenes, effectively handling intricate structures and overlapping objects.

Figures (8)

• Figure 1: Qualitative comparison with DepthAnything-V1 yang2024depth. Our method excels in capturing more complex visual elements and details.
• Figure 2: Overview of the proposed BriGeS framework. We design a pipeline that adaptively integrates semantic information into a depth foundation model through the Bridging Gate. During inference, this integration is further refined by Attention Temperature Scaling.
• Figure 3: Qualitative results on unseen datasets of BriGeS (Ours) and various state-of-the-art methods. BriGeS is highly effective at capturing intricate structures and delicate objects, such as thin wires, tree branches, and nets. Best viewed when zoomed in.
• Figure 4: Qualitative results of ablation study on proposed modules. A reversed color map is applied for better visualization.
• Figure 5: Qualitative results on KITTI geiger2012we.