The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models
Yuhuan You, Lai Wei, Xihong Wu, Tianshu Qu
TL;DR
This paper addresses the lack of spatial understanding in large audio-language models by introducing The World is Not Mono (TWNM) and an Auditory Scene Analysis (ASA) framework with three hierarchical layers: Static Identification, Relational Integration, and Cognitive Reasoning. It proposes a synthetic binaural data pipeline, a Hybrid Feature Projector with dense fusion to decouple and fuse semantic and spatial streams, and a progressive curriculum from representation learning to policy optimization, including Group Relative Policy Optimization (GRPO). The approach yields robust spatial reasoning on a dedicated benchmark, significantly improving over baselines in L2-L3 tasks and demonstrating the feasibility of endowing LALMs with holistic acoustic scene intelligence. This work advances spatial perception as a core dimension for embodied, context-aware auditory AI with potential impacts on robotics, AR/VR, and assistive technologies, and outlines directions for generalization to wild recordings and multi-channel formats like Ambisonics.
Abstract
Existing large audio-language models perceive the world as "mono" -- a single stream of audio that ignores the critical spatial dimension ("where") required for universal acoustic scene analysis. To bridge this gap, we first introduce a hierarchical framework for Auditory Scene Analysis (ASA). Guided by this framework, we introduce a system that enables models like Qwen2-Audio to understand and reason about the complex acoustic world. Our framework achieves this through three core contributions: First, we build a large-scale, synthesized binaural audio dataset to provide the rich spatial cues. Second, we design a hybrid feature projector, which leverages parallel semantic and spatial encoders to extract decoupled representations. These distinct streams are integrated via a dense fusion mechanism, ensuring the model receives a holistic view of the acoustic scene. Finally, we employ a progressive training curriculum, advancing from supervised fine-tuning (SFT) to reinforcement learning via Group Relative Policy Optimization (GRPO), to explicitly evolve the model's capabilities towards reasoning. On our comprehensive benchmark, the model demonstrates comparatively strong capability for spatial understanding. By enabling this spatial perception, our work provides a clear pathway for leveraging the powerful reasoning abilities of large models towards holistic acoustic scene analysis, advancing from "mono" semantic recognition to spatial intelligence.