SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation

TL;DR

SpatialBench-UC tackles the challenge of evaluating whether text-to-image models follow explicit spatial prompts under uncertainty. It introduces an uncertainty-aware evaluation framework that outputs PASS/FAIL/UNDECIDABLE with calibrated confidence, alongside a reproducible benchmark consisting of 200 prompts and 100 counterfactual pairs. The workflow combines detector-based grounding with geometry checks, four-component confidence, and abstention to produce risk–coverage analyses, augmented by a lightweight human audit for calibration. Grounding methods (BoxDiff, GLIGEN) substantially improve both pass rate and coverage compared to prompt-only generation, while abstention is still dominated by missing detections, highlighting detector limitations as a key bottleneck. Overall, SpatialBench-UC provides a transparent, auditable, and extensible platform for reproducible comparisons of spatial prompt following in diffusion models, with practical implications for robust evaluation and model development.

Abstract

Evaluating whether text-to-image models follow explicit spatial instructions is difficult to automate. Object detectors may miss targets or return multiple plausible detections, and simple geometric tests can become ambiguous in borderline cases. Spatial evaluation is naturally a selective prediction problem, the checker may abstain when evidence is weak and report confidence so that results can be interpreted as a risk coverage tradeoff rather than a single score. We introduce SpatialBench-UC, a small, reproducible benchmark for pairwise spatial relations. The benchmark contains 200 prompts (50 object pairs times 4 relations) grouped into 100 counterfactual pairs obtained by swapping object roles. We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables, enabling reproducible and auditable comparisons across models. We also include a lightweight human audit used to calibrate the checker's abstention margin and confidence threshold. We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN. The checker reports pass rate and coverage as well as conditional pass rates on decided samples. The results show that grounding methods substantially improve both pass rate and coverage, while abstention remains a dominant factor due mainly to missing detections.

Figures (10)

• Figure 1: Qualitative examples of checker outcomes. Each panel shows a generated image with detector boxes (red=A, blue=B) and the checker verdict. We include one clear PASS, one clear FAIL, and one abstention example where geometry is near the decision boundary.
• Figure 2: Pipeline overview. The evaluator outputs PASS/FAIL/UNDECIDABLE with confidence, enabling reporting under abstention; a small human audit calibrates parameters and operating points.
• Figure 3: Risk--coverage on the audited subset for the calibrated checker (sweep over unique confidence values). The curve is step-like because coverage changes only when $\tau$ crosses a sample’s discrete confidence value. Higher confidence thresholds reduce risk but also reduce coverage; risk excludes human-UNDECIDABLE labels from the accuracy denominator.
• Figure 4: Coverage vs conditional PASS (calibrated report). Higher coverage means fewer abstentions; conditional PASS is computed only on decided samples (PASS/FAIL).
• Figure 5: Qualitative support: high-confidence PASS examples. Overlays show detected boxes and the checker verdict for the specified spatial relation.