On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation

TL;DR

The paper challenges the presumed benefits of language priors for low-level monocular depth estimation by systematically evaluating how scene-level vs. low-level language descriptions affect robustness and generalization. Using the VPD framework and a suite of sentence-types that encode spatial relationships, the authors show a strong scene-level bias: depth models perform best with scene-level prompts and deteriorate when low-level relational language is introduced, even under zero-shot conditions. Through targeted analyses of supervised vs. zero-shot settings, adversarial masking, and schema-driven scene shifts, the study reveals notable failures in current language-guided depth estimation and attributes them to cross-modal grounding limitations in diffusion-based encoders and CLIP’s limited spatial grounding. The work provides a concrete framework for constructing low-level language inputs, highlights failure modes, and calls for robust grounding and distributionally aware designs to make language-guided depth estimation viable in real-world, safety-critical applications.

Abstract

Recent advances in monocular depth estimation have been made by incorporating natural language as additional guidance. Although yielding impressive results, the impact of the language prior, particularly in terms of generalization and robustness, remains unexplored. In this paper, we address this gap by quantifying the impact of this prior and introduce methods to benchmark its effectiveness across various settings. We generate "low-level" sentences that convey object-centric, three-dimensional spatial relationships, incorporate them as additional language priors and evaluate their downstream impact on depth estimation. Our key finding is that current language-guided depth estimators perform optimally only with scene-level descriptions and counter-intuitively fare worse with low level descriptions. Despite leveraging additional data, these methods are not robust to directed adversarial attacks and decline in performance with an increase in distribution shift. Finally, to provide a foundation for future research, we identify points of failures and offer insights to better understand these shortcomings. With an increasing number of methods using language for depth estimation, our findings highlight the opportunities and pitfalls that require careful consideration for effective deployment in real-world settings

Figures (12)

• Figure 1: We investigate the efficacy of language guidance for depth estimation by evaluating the robustness, generalization, and spurious biases associated with this approach, comparing it alongside traditional vision-only methods. Shown here is a visual comparison of the depth estimation results between VPD (with additional knowledge) and AdaBins Farooq_Bhat_2021 on an out-of-domain outdoor scene.
• Figure 2: An illustration of depth maps generated by language-guided depth estimation methods such as VPD (zero-shot) when prompted with various sentence inputs that we use as part of our study. The first row shows the effect of progressively adding descriptions as input, while the second row shows depth maps generated by single sentence inputs.
• Figure 3: We systematically create additional knowledge for the depth estimator by leveraging intrinsic and low-level image properties. For each image we derive scene addendums, object and spatial level sentences along with semantic, activity based descriptions, and in supervised and zero-shot settings, quantify the effect of these sentences on monocular depth estimation.
• Figure 4: Comparison of depth maps across the three models trained under the supervised setting as described in Table \ref{['tab:supervised_Setting']}. Low-level sentences induce hallucinations in the model; leading to large errors and false positive long-range depth estimates
• Figure 5: From left to right, as more bottom-up scene-level information is provided, the model's depth predictions move closer to the baseline predictions made with scene-level sentences. The plot below shows performance improvement across all metrics.