Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAVGBench: Benchmarking Spatially Aligned Audio-Video Generation

Kazuki Shimada, Christian Simon, Takashi Shibuya, Shusuke Takahashi, Yuki Mitsufuji

TL;DR

This work defines SAVGBench, a benchmark for Spatially Aligned Audio-Video Generation, to address the lack of multimodal models that produce videos with spatially consistent audio. It introduces a STARSS23-derived dataset of perspective video and stereo audio focusing on onscreen sound events, along with a novel Spatial AV-Align metric that evaluates spatial synchronization without requiring ground-truth audio. By evaluating a joint diffusion baseline (Stereo MM-Diffusion) and a two-stage pipeline (Video Diffusion + Stereo MMAudio), the study reveals meaningful gaps to ground truth in both audiovisual quality and spatial alignment, with the joint method showing better alignment than the two-stage approach. The benchmark and metrics provide a foundation for advancing truly spatially coherent audio-visual generation and point to future directions such as expanding environments, diversifying sound events, and exploring text-conditioned SAVG.

Abstract

This work addresses the lack of multimodal generative models capable of producing high-quality videos with spatially aligned audio. While recent advancements in generative models have been successful in video generation, they often overlook the spatial alignment between audio and visuals, which is essential for immersive experiences. To tackle this problem, we establish a new research direction in benchmarking the Spatially Aligned Audio-Video Generation (SAVG) task. We introduce a spatially aligned audio-visual dataset, whose audio and video data are curated based on whether sound events are onscreen or not. We also propose a new alignment metric that aims to evaluate the spatial alignment between audio and video. Then, using the dataset and metric, we benchmark two types of baseline methods: one is based on a joint audio-video generation model, and the other is a two-stage method that combines a video generation model and a video-to-audio generation model. Our experimental results demonstrate that gaps exist between the baseline methods and the ground truth in terms of video and audio quality, as well as spatial alignment between the two modalities.

SAVGBench: Benchmarking Spatially Aligned Audio-Video Generation

TL;DR

This work defines SAVGBench, a benchmark for Spatially Aligned Audio-Video Generation, to address the lack of multimodal models that produce videos with spatially consistent audio. It introduces a STARSS23-derived dataset of perspective video and stereo audio focusing on onscreen sound events, along with a novel Spatial AV-Align metric that evaluates spatial synchronization without requiring ground-truth audio. By evaluating a joint diffusion baseline (Stereo MM-Diffusion) and a two-stage pipeline (Video Diffusion + Stereo MMAudio), the study reveals meaningful gaps to ground truth in both audiovisual quality and spatial alignment, with the joint method showing better alignment than the two-stage approach. The benchmark and metrics provide a foundation for advancing truly spatially coherent audio-visual generation and point to future directions such as expanding environments, diversifying sound events, and exploring text-conditioned SAVG.

Abstract

This work addresses the lack of multimodal generative models capable of producing high-quality videos with spatially aligned audio. While recent advancements in generative models have been successful in video generation, they often overlook the spatial alignment between audio and visuals, which is essential for immersive experiences. To tackle this problem, we establish a new research direction in benchmarking the Spatially Aligned Audio-Video Generation (SAVG) task. We introduce a spatially aligned audio-visual dataset, whose audio and video data are curated based on whether sound events are onscreen or not. We also propose a new alignment metric that aims to evaluate the spatial alignment between audio and video. Then, using the dataset and metric, we benchmark two types of baseline methods: one is based on a joint audio-video generation model, and the other is a two-stage method that combines a video generation model and a video-to-audio generation model. Our experimental results demonstrate that gaps exist between the baseline methods and the ground truth in terms of video and audio quality, as well as spatial alignment between the two modalities.

Paper Structure

This paper contains 15 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SAVGBench
  4. Dataset Overview
  5. Data Construction
  6. Evaluation Metrics
  7. Baseline Methods
  8. Joint Method: Stereo MM-Diffusion
  9. Two-stage Method: Video Diffusion and Stereo MMAudio
  10. Experimental Evaluation
  11. Experimental Setup
  12. Experimental Result
  13. Subjective Test
  14. Conclusion
  15. Acknowledgments

Figures (2)

  • Figure 1: Examples of our proposed dataset. It features lectures, conversations, and playing musical instruments in indoor environments such as modern classrooms or meeting rooms. Perspective videos are square-shaped with padding. Stereo audio is displayed as a two-channel spectrogram.
  • Figure 2: An illustration of the detected object (person class) and the detected sound event (instrument class) in the Spatial AV-Align metric. Padding is omitted for brevity. The green box indicates the detected object using the object detector. The blue box indicates the detected sound event using the SELD model. The SELD result has a margin around the estimated horizontal position. Its vertical range is set from top to bottom as it does not estimate a vertical position.