GRAM: Spatial general-purpose audio representations for real-world environments

TL;DR

GRAM introduces a multi-channel masked autoencoder to learn spatial audio representations from binaural and Ambisonics inputs, addressing limitations of existing models in reverberant, noisy real-world environments. By simulating 85,000 realistic sound scenes with SoundSpaces 2.0 and training with a mixed real-world/dry data pipeline, GRAM achieves state-of-the-art performance on HEAR and NatHEAR benchmarks and demonstrates strong SELD transfer in RealSELD. The model preserves spatial cues such as ILD and IV and uses a patch-based MAE with local-global attention to reconstruct masked spectrogram patches, enabling localization and scene analysis tasks. The NatHEAR and RealSELD benchmarks provide a standardized platform for evaluating spatial audio embeddings in both simulated and real-world environments. Overall, GRAM advances robust, spatially-aware foundation models for real-world acoustic scenes and enables efficient transfer to real recordings.

Abstract

Audio foundation models learn general-purpose audio representations that facilitate a wide range of downstream tasks. While the performance of these models has greatly increased for conventional single-channel, dry audio clips, their success in real-world acoustic environments with reverberation and noise is limited. Furthermore, most audio foundation models ignore the spatial dimension of real-world acoustic environments, ruling out tasks involving sound localization. To address these limitations, we propose GRAM: a general-purpose real-world audio model that employs a multi-channel masked autoencoder to efficiently learn spatial audio representations. We evaluated GRAM and other audio foundation models in a standardized manner on high-quality simulations of naturalistic, spatial acoustic environments as well as recordings of real-world environments and release these two complementary benchmark task suites: NatHEAR and RealSELD. Our results demonstrate that GRAM outperforms all state-of-the-art self-supervised audio foundation models on NatHEAR and the clean, single-channel version HEAR, while using only a fraction of the training data. GRAM also shows state-of-the-art localization performance in simulated environments and generalizes efficiently to real-world recordings in RealSELD. Taken together, GRAM presents a significant advance toward robust spatial audio foundation models for real-world environments.

Figures (8)

• Figure 1: Proposed self-supervised approach for training GRAMs on naturalistic binaural scenes. (A) We generate binaural and ambisonics naturalistic scenes using the SoundSpace2.0 simulator chen22soundspaces2 in MatterPort3D houses. (B) MAE approach for learning audio representations with spatial attributes. For the ambisonics spectrograms, the methodology remains the same, except that the inputs now include 4-channel mel-spectrograms and intensity vectors (IVs).
• Figure 2: Downstream model performance. (A) NatHEAR and HEAR performance as a function of training data. (B) Difference in performance on HEAR and NatHEAR (excluding the DCASE-2016 task). Box limits reflect first and third quartile, center line the median.
• Figure 3: Sound localization and T60 estimation in simulated real-world sound scenes. (A) Direction of arrival (DoA) error. (B) T60 estimation absolute error. Boxes indicate first and third quartile; center line: median; whiskers: 1.5 times the interquartile range.
• Figure 4: Ablation studies. Scores reflect downstream task performance across all tasks of the benchmark task suite (y-axis, $s(m)$). From left to right: (1) Impact of mixing clean and naturalistic scenes during pre-training of GRAM-Binaural. (2) Impact of ratio $\lambda$ of clean and naturalistic scenes during pre-training of GRAM-Ambisonics. (3) Effect of masking strategy for GRAM-Binaural (4) Comparing Mamba and Transformer encoders for binaural audio.
• Figure 5: Impact of real-life data during training. Comparing GRAM-Ambisonics validation scores on STARSS23 for training from scratch, fine-tuning a pre-trained model, and direct transfer of the pre-trained model using HEAR pipeline.