DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Models
Kevin Wilkinghoff, Zheng-Hua Tan
TL;DR
DSpAST introduces a disentangled spatial audio encoder that extends SpatialAST with per-task branches for SED, distance prediction, and direction-of-arrival estimation, while sharing the transformer backbone to maintain a comparable parameter count. By adding extra spatial features and a per-task feature attention module, DSpAST learns task-specific representations that improve performance on spatial audio reasoning without a large model size increase. Empirical results show DSPAST outperforms SpatialAST on binaural datasets and yields better end-to-end performance when used as a tokenizer in the BAT-based SpatialSoundQA system, with single-stage training preferred for efficiency. The work also provides interpretability through attention weight analysis and outlines opportunities for further enhancements in feature engineering, multi-microphone setups, and cross-modal integration.
Abstract
Reasoning about spatial audio with large language models requires a spatial audio encoder as an acoustic front-end to obtain audio embeddings for further processing. Such an encoder needs to capture all information required to detect the type of sound events, as well as the direction and distance of their corresponding sources. Accomplishing this with a single audio encoder is demanding as the information required for each of these tasks is mostly independent of each other. As a result, the performance obtained with a single encoder is often worse than when using task-specific audio encoders. In this work, we present DSpAST, a novel audio encoder based on SpatialAST that learns disentangled representations of spatial audio while having only 0.2% additional parameters. Experiments on SpatialSoundQA with the spatial audio reasoning system BAT demonstrate that DSpAST significantly outperforms SpatialAST.