Visual-based spatial audio generation system for multi-speaker environments

TL;DR

This work presents a visual-based spatial audio generation system that automates audio-visual alignment in multi-speaker environments without requiring binaural datasets. By integrating YOLOv8-based object detection, monocular depth estimation from Depth Anything, and dual spatialization paths (HRTF and a 3D algorithm), the method converts mono audio into spatial audio guided by visual cues. Conv-TasNet (with Demucs for music) performs source separation, while visual coordinates drive 3D positioning and acoustic rendering, yielding improved spatial consistency and speech quality across datasets, including challenging multi-speaker scenarios. The approach offers a practical, efficient tool for post-production professionals to streamline spatial audio workflows with robust cross-modal performance.

Abstract

In multimedia applications such as films and video games, spatial audio techniques are widely employed to enhance user experiences by simulating 3D sound: transforming mono audio into binaural formats. However, this process is often complex and labor-intensive for sound designers, requiring precise synchronization of audio with the spatial positions of visual components. To address these challenges, we propose a visual-based spatial audio generation system - an automated system that integrates face detection YOLOv8 for object detection, monocular depth estimation, and spatial audio techniques. Notably, the system operates without requiring additional binaural dataset training. The proposed system is evaluated against existing Spatial Audio generation system using objective metrics. Experimental results demonstrate that our method significantly improves spatial consistency between audio and video, enhances speech quality, and performs robustly in multi-speaker scenarios. By streamlining the audio-visual alignment process, the proposed system enables sound engineers to achieve high-quality results efficiently, making it a valuable tool for professionals in multimedia production.

Figures (4)

• Figure 1: Spatialisation of audio representing X, Y and Z coordinates.
• Figure 2: Flowgraph of the system showcasing the main processing pipeline.
• Figure 3: YOLOv8-n Output: Object Detection with Bounding Box Predictions
• Figure 4: Depth estimation model visualization.