SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis

TL;DR

SANEval addresses the lack of open-world, fine-grained evaluation for text-to-image models by introducing a compositional benchmark with open-vocabulary capabilities. It combines a Prompt Understanding Module (driven by LLMs) and an Enhanced Object Detection Module (open-world detector with synonym expansion) to assess attribute binding, spatial relationships, and numeracy, producing both numeric scores and actionable diagnostic feedback. The benchmark is validated across six state-of-the-art T2I models and shows strong correlation with human judgments, while revealing robustness to different LLM backbones and highlighting areas such as shape binding that remain challenging. By releasing the dataset and an open-source evaluation pipeline, SANEval enables feedback-driven, open-world benchmarking to guide future improvements in compositional generation and evaluation.

Abstract

The rapid progress of text-to-image (T2I) models has unlocked unprecedented creative potential, yet their ability to faithfully render complex prompts involving multiple objects, attributes, and spatial relationships remains a significant bottleneck. Progress is hampered by a lack of adequate evaluation methods; current benchmarks are often restricted to closed-set vocabularies, lack fine-grained diagnostic capabilities, and fail to provide the interpretable feedback necessary to diagnose and remedy specific compositional failures. We solve these challenges by introducing SANEval (Spatial, Attribute, and Numeracy Evaluation), a comprehensive benchmark that establishes a scalable new pipeline for open-vocabulary compositional evaluation. SANEval combines a large language model (LLM) for deep prompt understanding with an LLM-enhanced, open-vocabulary object detector to robustly evaluate compositional adherence, unconstrained by a fixed vocabulary. Through extensive experiments on six state-of-the-art T2I models, we demonstrate that SANEval's automated evaluations provide a more faithful proxy for human assessment; our metric achieves a Spearman's rank correlation with statistically different results than those of existing benchmarks across tasks of attribute binding, spatial relations, and numeracy. To facilitate future research in compositional T2I generation and evaluation, we will release the SANEval dataset and our open-source evaluation pipeline.

Figures (12)

• Figure 1: Block diagram description of the SANEval benchmark
• Figure 2: Block diagram of the enhanced object detection (OD) pipeline for the SANEval benchmark.
• Figure 3: Example of the Attribute Binding evaluation pipeline in SANEval. Objects are first detected and cropped, then evaluated by a VLM on the correctness of their attributes. The upper example fully adheres to the prompt (“pink stop sign and orange bird”), yielding a perfect score of 1.0 with no feedback. The lower example partially fails (“yellow peach and a lime ceiling”), where the ceiling is missing, resulting in diagnostic feedback and a lower score of 0.5
• Figure 4: Block diagram descriptions of the three different scorers for the SANEval benchmark.
• Figure 5: Human evaluation interface used to collect adherence responses for spatial dataset.