SPACE-CLIP: Spatial Perception via Adaptive CLIP Embeddings for Monocular Depth Estimation

TL;DR

SPACE-CLIP reframes monocular depth estimation as direct interpretation of latent geometry from a frozen CLIP vision encoder, avoiding text prompts and backbone fine-tuning. It introduces a Dense Predictor with a Semantic Pathway (FiLM-conditioned high-level features) and a Structural Pathway (low-level details), fused hierarchically to produce high-fidelity depth maps. Across KITTI, the approach achieves competitive performance with a clear ablation showing the necessity of both pathways, and demonstrates a practical, modular perception plugin for embodied AI like VLA models. This work offers a scalable blueprint for augmenting foundation models with spatial reasoning while preserving efficiency and integration ease for real-world robotic and vision-language systems.

Abstract

Contrastive Language-Image Pre-training (CLIP) has accomplished extraordinary success for semantic understanding but inherently struggles to perceive geometric structure. Existing methods attempt to bridge this gap by querying CLIP with textual prompts, a process that is often indirect and inefficient. This paper introduces a fundamentally different approach using a dual-pathway decoder. We present SPACE-CLIP, an architecture that unlocks and interprets latent geometric knowledge directly from a frozen CLIP vision encoder, completely bypassing the text encoder and its associated textual prompts. A semantic pathway interprets high-level features, dynamically conditioned on global context using feature-wise linear modulation (FiLM). In addition, a structural pathway extracts fine-grained spatial details from early layers. These complementary streams are hierarchically fused, enabling a robust synthesis of semantic context and precise geometry. Extensive experiments on the KITTI benchmark show that SPACE-CLIP dramatically outperforms previous CLIP-based methods. Our ablation studies validate that the synergistic fusion of our dual pathways is critical to this success. SPACE-CLIP offers a new, efficient, and architecturally elegant blueprint for repurposing large-scale vision models. The proposed method is not just a standalone depth estimator, but a readily integrable spatial perception module for the next generation of embodied AI systems, such as vision-language-action (VLA) models. Our model is available at https://github.com/taewan2002/space-clip

Figures (3)

• Figure 1: The VLA Integration Challenge and the SPACE-CLIP Solution. This figure illustrates the architectural paradigm shift proposed by our work. (a) The Inefficient Path: Conventional CLIP-based methods integrate depth estimation as an external module, creating two major bottlenecks for a Vision-Language-Action (VLA) model. (1) Architectural Conflict arises from needing a separate, inefficient data path from the VLA's primary vision encoder. (2) Textual Interference occurs as the VLA's reasoning engine must generate a textual query, conflicting with its main language tasks. (b) SPACE-CLIP's Seamless Integration: Our model acts as a lightweight, decoder-only module that attaches directly to the VLA's existing, frozen vision encoder. By directly interpreting hierarchical features, it provides spatial awareness without architectural redundancy or textual interference, enabling a truly integrated and efficient agent.
• Figure 2: The overall architecture of SPACE-CLIP. Our model leverages a frozen CLIP vision encoder to extract multi-level features, which are processed through a novel Dual Pathway architecture within the Dense Predictor. (1) The Semantic Decoder (top path) processes high-level features, dynamically modulated by the global context via a FiLM layer. (2) The Structural Decoder (bottom path) processes low-level features to preserve fine-grained details. The outputs of these two pathways are hierarchically fused at each upsampling stage. This fusion mechanism allows SPACE-CLIP to generate semantically coherent and geometrically precise depth maps.
• Figure 3: Qualitative comparison on the KITTI dataset. From left to right: Input Image, Ground Truth, and our SPACE-CLIP's prediction. Our model generates dense and detailed depth maps that accurately capture complex structures such as thin poles, distant vehicles, and foliage, demonstrating its superior geometric understanding.