Predicting When to Trust Vision-Language Models for Spatial Reasoning

TL;DR

This work tackles the poor spatial reasoning of vision-language models by introducing a vision-based confidence estimator that uses external geometric verification via object detection to validate spatial predictions. Four complementary signals—geometric alignment, separation, detection quality, and VLM token confidence—are fused with XGBoost to predict the likelihood that a VLM output is correct, enabling selective prediction. Across BLIP-2 and CLIP on the Visual Spatial Reasoning benchmark, the method achieves substantial AUROC gains (0.674 for BLIP-2 and 0.583 for CLIP) and higher coverage at target accuracies, while vision-based features dominate model importance. The approach supports practical scene-graph construction and offers a scalable, efficient path toward deploying VLMs in robotics and other safety-critical tasks by prioritizing external verification over self-assessment.

Abstract

Vision-Language Models (VLMs) demonstrate impressive capabilities across multimodal tasks, yet exhibit systematic spatial reasoning failures, achieving only 49% (CLIP) to 54% (BLIP-2) accuracy on basic directional relationships. For safe deployment in robotics and autonomous systems, we need to predict when to trust VLM spatial predictions rather than accepting all outputs. We propose a vision-based confidence estimation framework that validates VLM predictions through independent geometric verification using object detection. Unlike text-based approaches relying on self-assessment, our method fuses four signals via gradient boosting: geometric alignment between VLM claims and coordinates, spatial ambiguity from overlap, detection quality, and VLM internal uncertainty. We achieve 0.674 AUROC on BLIP-2 (34.0% improvement over text-based baselines) and 0.583 AUROC on CLIP (16.1% improvement), generalizing across generative and classification architectures. Our framework enables selective prediction: at 60% target accuracy, we achieve 61.9% coverage versus 27.6% baseline (2.2x improvement) on BLIP-2. Feature analysis reveals vision-based signals contribute 87.4% of model importance versus 12.7% from VLM confidence, validating that external geometric verification outperforms self-assessment. We demonstrate reliable scene graph construction where confidence-based pruning improves precision from 52.1% to 78.3% while retaining 68.2% of edges.

Figures (5)

• Figure 1: Overview of our vision-based confidence estimation pipeline. Given a VLM's spatial prediction, we validate it through geometric verification using object detection. Four complementary confidence signals (geometric alignment, object separation, detection quality, VLM internal confidence) are fused via gradient boosting to produce a final confidence score. High-confidence predictions (green) are trusted; low-confidence predictions (red) are rejected or deferred to human review.
• Figure 2: Coverage vs. target accuracy for BLIP-2 (left) and CLIP (right). Our method (solid blue) achieves 2.2× improvement over geometric baseline (dashed orange) at 60% accuracy, demonstrating effective discrimination of reliable predictions.
• Figure 3: Precision-recall curves comparing our method (filled squares) against Khan et al. khan2023consistency (hollow circles) for BLIP-2 (left) and CLIP (right). Khan's text-based method exhibits extreme behaviors—BLIP-2 trusts nearly everything (100% recall, 51.6% precision), while CLIP rejects almost everything (1.8% recall, approaching the origin). Our vision-based approach achieves balanced precision-recall tradeoffs, demonstrating effective error discrimination through geometric validation.
• Figure 4: Feature importance analysis for BLIP-2. Geometric confidence dominates at 37.5%, validating that coordinate-based spatial verification provides the strongest signal. Separation confidence (32.7%) captures ambiguity from overlapping objects, detection quality (17.2%) indicates visual difficulty, and VLM token confidence (12.7%) provides internal uncertainty. Vision-based features collectively account for 87.4% of model importance.
• Figure 5: Representative test examples illustrating method behavior. (a) High-confidence agreement: VLM and geometric validation agree (both predict "left"), method correctly trusts with high confidence (0.95). (b) Contradiction detected (key advantage): VLM predicts "left" but geometry indicates "right"—our method correctly rejects (0.08) while baseline incorrectly trusts (0.59), demonstrating effective error detection. (c) Detection failure (limitation): Object detection fails (Geo: none)—method expresses moderate uncertainty (0.46) as geometric validation is unavailable. (d) Appropriate uncertainty: Another detection failure case where method correctly expresses low confidence (0.33), demonstrating robustness when evidence is insufficient.