Assessing Situational and Spatial Awareness of VLMs with Synthetically Generated Video

TL;DR

Assessing Situational and Spatial Awareness of VLMs with Synthetically Generated Video addresses the fragility of spatio-temporal reasoning in vision–language systems by introducing a synthetic, minimal-pair video benchmark. The method uses 120-frame clips rendered in Unreal Engine with controlled variables to test situational awareness, role binding across viewpoints, and fine-grained trajectory alignment, evaluated in a training-free setting with a lightweight text classifier to map outputs to labels. Key findings show performance near chance across tasks, with color-based identifiers partially mitigating role-binding errors but not resolving core weaknesses, underscoring the need for explicit spatial grounding. The work provides reproducible diagnostics and suggests lightweight spatial priors to complement large-scale pretraining, with data/code released for future research and robust evaluation under varied viewpoints and occlusions.

Abstract

Spatial reasoning in vision language models (VLMs) remains fragile when semantics hinge on subtle temporal or geometric cues. We introduce a synthetic benchmark that probes two complementary skills: situational awareness (recognizing whether an interaction is harmful or benign) and spatial awareness (tracking who does what to whom, and reasoning about relative positions and motion). Through minimal video pairs, we test three challenges: distinguishing violence from benign activity, binding assailant roles across viewpoints, and judging fine-grained trajectory alignment. While we evaluate recent VLMs in a training-free setting, the benchmark is applicable to any video classification model. Results show performance only slightly above chance across tasks. A simple aid, stable color cues, partly reduces assailant role confusions but does not resolve the underlying weakness. By releasing data and code, we aim to provide reproducible diagnostics and seed exploration of lightweight spatial priors to complement large-scale pretraining.

Figures (6)

• Figure 1: Evaluation pipeline. Image models receive uniformly sampled frames; video models receive the full clip. Free-form generations are mapped to the label set by a lightweight text classifier.
• Figure 2: Example scenes used in experiments. The rightmost column uses the HDRI Buikslotermeerplein for global illumination variation while preserving geometry.
• Figure 3: Situational discrimination (fighting vs. dancing/idle). Confusion matrices by model indicate reliable separation under controlled appearance and backgrounds.
• Figure 4: Role binding with and without stable color identifiers. Color substantially reduces left/right errors and role-flip sensitivity, isolating a linguistic anchoring failure.
• Figure 5: Impact of requiring explanations across experiments: beneficial for models with sparse outputs (e.g., Gemma) but detrimental for others (e.g., Qwen2.5-VL)