R-AVST: Empowering Video-LLMs with Fine-Grained Spatio-Temporal Reasoning in Complex Audio-Visual Scenarios

TL;DR

<p>R-AVST introduces the first video dataset tailored for fine-grained spatio-temporal reasoning in realistic audio-visual scenes, addressing gaps in existing AV datasets by coupling auditory and visual cues with precise temporal boundaries and spatial localization. The authors propose three targeted reasoning tasks and automatically generated QA pairs, enabling robust evaluation of Video-LLMs in complex scenes. To push model capability, they develop AVST-Zero, a GRPO-based Video-LLM trained on R-AVST with a multi-dimensional reward design (format, object, temporal, spatial), achieving competitive performance across tasks and highlighting the benefits of reinforcement learning for fine-grained AV reasoning. The work combines a scalable data-generation pipeline (caption analysis, Grounded-SAM2 grounding, QA generation) with an RL-driven training paradigm, offering a valuable benchmark and a viable path toward more capable audio-visual reasoning systems.</p>

Abstract

Recently, rapid advancements have been made in multimodal large language models (MLLMs), especially in video understanding tasks. However, current research focuses on simple video scenarios, failing to reflect the complex and diverse nature of real-world audio-visual events in videos. To bridge this gap, we firstly introduce R-AVST, a dataset for audio-visual reasoning featuring fine-grained spatio-temporal annotations. In constructing this, we design a pipeline consisting of LLM-based key object extraction, automatic spatial annotation and manual quality inspection, resulting in over 5K untrimmed videos with 27K objects across 100 types of audio-visual events. Building on this dataset, we define three core tasks for spatio-temporal reasoning in audio-visual scenes and generate more than 8K high-quality, evenly distributed question-answer pairs to effectively benchmark model performance. To further enhance reasoning, we propose AVST-Zero, a reinforcement learning-based model that avoids intermediate supervision, directly optimizing behavior via carefully designed multi-dimensional rewards. Extensive experiments validate the effectiveness of our R-AVST in advancing audio-visual spatio-temporal reasoning, upon which AVST-Zero demonstrates competitive performance compared to existing models. To the best of our knowledge, R-AVST is the first dataset designed for real-world audio-visual spatio-temporal reasoning, and AVST-Zero offers a novel perspective for tackling future challenges in this domain.

Figures (6)

• Figure 1: Unlike previous datasets, R-AVST focuses on spatio-temporal reasoning in complex audio-visual scenes of untrimmed videos, offering fine-grained temporal boundary and spatial localization annotations. This example shows three core tasks designed to evaluate reasoning over sounding objects, time, and space.
• Figure 2: Data generation pipeline of R-AVST. The tasks are explicitly designed to capture both spatial and temporal aspects of complex audio-visual scenes. The dataset construction follows a five-step process, yielding fine-grained spatio-temporal annotations and task-oriented QAs.
• Figure 3: Statistics of R-AVST dataset. (a) Duration distribution of different event categories in descending order, where colors represent their corresponding coarse-grained categories. (b) Duration distribution of all videos. (c) Distribution of the number of audio-visual events per video. (d) Word cloud of event categories.
• Figure 4: Model architecture of our AVST-Zero model. The multi-dimensional reward design allows AVST-Zero to perform exceptionally well in spatio-temporal reasoning tasks.
• Figure 5: Qualitative results. For the bounding boxes in the video: green denotes the ground truth, blue comes from VideoChat-R1, yellow from Qwen2.5-VL, and red from our AVST-Zero.