STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence

TL;DR

STAR-Bench defines audio 4D intelligence as deep spatio-temporal reasoning over sound dynamics and introduces a two-tier benchmark: Foundational Acoustic Perception for precise perceptual acuity, and Holistic Spatio-Temporal Reasoning for multi-source, temporally and spatially consistent inference. The data pipeline combines procedurally synthesized audio with real-world corpora, plus rigorous human annotation and validation to ensure high-quality questions. Evaluation across 19 models reveals a substantial gap to humans and highlights a capability split between closed- and open-source systems, with spatial reasoning being particularly challenging due to multi-channel processing requirements. The work provides actionable diagnostics and a clear path forward for developing models with robust, physically grounded auditory understanding applicable to embodied AI.

Abstract

Despite rapid progress in Multi-modal Large Language Models and Large Audio-Language Models, existing audio benchmarks largely test semantics that can be recovered from text captions, masking deficits in fine-grained perceptual reasoning. We formalize audio 4D intelligence that is defined as reasoning over sound dynamics in time and 3D space, and introduce STAR-Bench to measure it. STAR-Bench combines a Foundational Acoustic Perception setting (six attributes under absolute and relative regimes) with a Holistic Spatio-Temporal Reasoning setting that includes segment reordering for continuous and discrete processes and spatial tasks spanning static localization, multi-source relations, and dynamic trajectories. Our data curation pipeline uses two methods to ensure high-quality samples. For foundational tasks, we use procedurally synthesized and physics-simulated audio. For holistic data, we follow a four-stage process that includes human annotation and final selection based on human performance. Unlike prior benchmarks where caption-only answering reduces accuracy slightly, STAR-Bench induces far larger drops (-31.5\% temporal, -35.2\% spatial), evidencing its focus on linguistically hard-to-describe cues. Evaluating 19 models reveals substantial gaps compared with humans and a capability hierarchy: closed-source models are bottlenecked by fine-grained perception, while open-source models lag across perception, knowledge, and reasoning. Our STAR-Bench provides critical insights and a clear path forward for developing future models with a more robust understanding of the physical world.

Figures (19)

• Figure 1: (Left): A comparison between humans and the Gemini 2.5 Pro with and without audio captions on various audio benchmarks. Our STAR-Bench evaluates linguistically hard-to-describe audio cues. See \ref{['appendix:caption_prompt']} for audio caption details. (Right): The three core abilities required to solve tasks in the STAR-Bench benchmark.
• Figure 2: Data examples from STAR-Bench: (1) the foundational perception task (upper) and (2) the holistic spatio-temporal reasoning task, which includes both temporal reasoning (bottom left) and spatial reasoning (bottom right). Zoom in for the best view.
• Figure 3: Audio preprocessing in existing models results in the loss of dual-channel information.
• Figure 4: Data distribution across the foundation perception, temporal reasoning, and spatial reasoning three tasks.
• Figure 5: Statistics.