STATUS Bench: A Rigorous Benchmark for Evaluating Object State Understanding in Vision-Language Models

TL;DR

This work introduces STATUS Bench, a rigorous, state-change–focused benchmark for vision-language models, built on 404 hand-annotated quintuplets that pair images with fine-grained object-state descriptions and state-change text. It defines three interdependent tasks—object state identification (OSI), image retrieval (IR), and state change identification (SCI)—operating under static and dynamic scenarios, and introduces Rigorous Overall Accuracy (ROA) to measure cross-task consistency. Complementing the benchmark, STATUS Train provides 13 million semi-automatic training examples derived from Ego4D, enabling fine-tuning that markedly improves performance; eight SOTA VLMs are evaluated, with open-weight models largely at chance on ROA, while fine-tuning on STATUS Train yields results on par with some closed models like Gemini 2.0 Flash. The findings underscore the necessity of a specialized benchmark and data resource to drive progress in fine-grained object-state understanding in vision-language systems and suggest directions for architecture and training improvements to achieve more reliable, consistent reasoning about object states.

Abstract

Object state recognition aims to identify the specific condition of objects, such as their positional states (e.g., open or closed) and functional states (e.g., on or off). While recent Vision-Language Models (VLMs) are capable of performing a variety of multimodal tasks, it remains unclear how precisely they can identify object states. To alleviate this issue, we introduce the STAte and Transition UnderStanding Benchmark (STATUS Bench), the first benchmark for rigorously evaluating the ability of VLMs to understand subtle variations in object states in diverse situations. Specifically, STATUS Bench introduces a novel evaluation scheme that requires VLMs to perform three tasks simultaneously: object state identification (OSI), image retrieval (IR), and state change identification (SCI). These tasks are defined over our fully hand-crafted dataset involving image pairs, their corresponding object state descriptions and state change descriptions. Furthermore, we introduce a large-scale training dataset, namely STATUS Train, which consists of 13 million semi-automatically created descriptions. This dataset serves as the largest resource to facilitate further research in this area. In our experiments, we demonstrate that STATUS Bench enables rigorous consistency evaluation and reveal that current state-of-the-art VLMs still significantly struggle to capture subtle object state distinctions. Surprisingly, under the proposed rigorous evaluation scheme, most open-weight VLMs exhibited chance-level zero-shot performance. After fine-tuning on STATUS Train, Qwen2.5-VL achieved performance comparable to Gemini 2.0 Flash. These findings underscore the necessity of STATUS Bench and Train for advancing object state recognition in VLM research.

Figures (7)

• Figure 1: STATUS Bench for rigorously evaluating the ability of VLMs to understand object states. We provide quintuplets, each consisting of a image pair $\bm{(x^{a}, x^{b})}$, corresponding object state descriptions $\bm{(t^{a}, t^{b})}$ and a state change description $\bm{y}$ (top). On the graph structure derived from these quintuplets, we define three evaluation tasks (OSI, IR, and SCI) using five sub-graphs to rigorously evaluate consistency (bottom).
• Figure 2: Annotated image examples from STATUS Bench. Each example is a quintuplet consisting of two images $\bm{(x^{a}, x^{b})}$, corresponding object state descriptions $\bm{(t^{a}, t^{b})}$, and a state change description $\bm{y}$.
• Figure 3: Proposed rigorous consistency evaluation scheme over quintuplet data. (a) Quintuplet consists of an image pair $(\bm{x}^{a}, \bm{x}^{b})$, a state description pair $(\bm{t}^{a}, \bm{t}^{b})$, and a state change description $\bm{y}$. Edges represent dependency, where dashed lines indicate incorrect links. (b) OSI task assumes that a single image is observed. VLMs are tasked with selecting the correct description. (c) IR task assumes that a single state description is observed. VLMs are tasked with selecting the correct image. (d) SCI task assumes that two images are observed. VLMs are tasked with selecting the correct stat change description.
• Figure 4: STATUS Train is a dataset semi-automatically created from video and narration data of Ego4D. Eleven key frames are extracted for each narrated event and used to generate time-consistent object state descriptions.
• Figure 5: Example results. Paired predictions for OSI and IR, and a single prediction for SCI are shown for each model. Consistently correct predictions that contribute to higher ROA values are marked in green. $^{\dagger}$ indicates a fine-tuned model.