Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BAT: Learning to Reason about Spatial Sounds with Large Language Models

Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, David Harwath

TL;DR

BAT presents a pioneering framework that endows large language models with spatial audio reasoning by pairing a specialized spatial encoder, Spatial-AST, with LLaMA-2. A synthetic SpatialSoundQA dataset, built from AudioSet and SoundSpaces 2.0, enables joint training of perception and reasoning about 3-D sound scenes. The three-stage perception-to-reasoning curriculum and integration with LLaMA-adapter v2 yield strong performance on both spatial perception metrics (e.g., DoA, distance) and reasoning tasks, including complex multi-source scenarios. This work unlocks new potential for multimodal AI in embodied and immersive settings, while acknowledging limitations such as source count, sim-to-real transfer, and potential extensions to ambisonics and other modalities.

Abstract

Spatial sound reasoning is a fundamental human skill, enabling us to navigate and interpret our surroundings based on sound. In this paper we present BAT, which combines the spatial sound perception ability of a binaural acoustic scene analysis model with the natural language reasoning capabilities of a large language model (LLM) to replicate this innate ability. To address the lack of existing datasets of in-the-wild spatial sounds, we synthesized a binaural audio dataset using AudioSet and SoundSpaces 2.0. Next, we developed SpatialSoundQA, a spatial sound-based question-answering dataset, offering a range of QA tasks that train BAT in various aspects of spatial sound perception and reasoning. The acoustic front end encoder of BAT is a novel spatial audio encoder named Spatial Audio Spectrogram Transformer, or Spatial-AST, which by itself achieves strong performance across sound event detection, spatial localization, and distance estimation. By integrating Spatial-AST with LLaMA-2 7B model, BAT transcends standard Sound Event Localization and Detection (SELD) tasks, enabling the model to reason about the relationships between the sounds in its environment. Our experiments demonstrate BAT's superior performance on both spatial sound perception and reasoning, showcasing the immense potential of LLMs in navigating and interpreting complex spatial audio environments.

BAT: Learning to Reason about Spatial Sounds with Large Language Models

TL;DR

BAT presents a pioneering framework that endows large language models with spatial audio reasoning by pairing a specialized spatial encoder, Spatial-AST, with LLaMA-2. A synthetic SpatialSoundQA dataset, built from AudioSet and SoundSpaces 2.0, enables joint training of perception and reasoning about 3-D sound scenes. The three-stage perception-to-reasoning curriculum and integration with LLaMA-adapter v2 yield strong performance on both spatial perception metrics (e.g., DoA, distance) and reasoning tasks, including complex multi-source scenarios. This work unlocks new potential for multimodal AI in embodied and immersive settings, while acknowledging limitations such as source count, sim-to-real transfer, and potential extensions to ambisonics and other modalities.

Abstract

Spatial sound reasoning is a fundamental human skill, enabling us to navigate and interpret our surroundings based on sound. In this paper we present BAT, which combines the spatial sound perception ability of a binaural acoustic scene analysis model with the natural language reasoning capabilities of a large language model (LLM) to replicate this innate ability. To address the lack of existing datasets of in-the-wild spatial sounds, we synthesized a binaural audio dataset using AudioSet and SoundSpaces 2.0. Next, we developed SpatialSoundQA, a spatial sound-based question-answering dataset, offering a range of QA tasks that train BAT in various aspects of spatial sound perception and reasoning. The acoustic front end encoder of BAT is a novel spatial audio encoder named Spatial Audio Spectrogram Transformer, or Spatial-AST, which by itself achieves strong performance across sound event detection, spatial localization, and distance estimation. By integrating Spatial-AST with LLaMA-2 7B model, BAT transcends standard Sound Event Localization and Detection (SELD) tasks, enabling the model to reason about the relationships between the sounds in its environment. Our experiments demonstrate BAT's superior performance on both spatial sound perception and reasoning, showcasing the immense potential of LLMs in navigating and interpreting complex spatial audio environments.
Paper Structure (26 sections, 7 equations, 4 figures, 12 tables)

This paper contains 26 sections, 7 equations, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Spatial Audio
  4. Sound Event Localization and Detection
  5. Multimodal Large Language Models
  6. SpatialSoundQA
  7. Spatial Audio Generation
  8. Question-Answer Pair Generation
  9. Method
  10. Spatial Audio Encoder: Spatial-AST
  11. Front-End.
  12. Bat: A Model for Reasoning about Spatial Sounds
  13. Experiments
  14. Implementation Details
  15. Performance of Spatial-AST on SELD Tasks
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Illustration of the Bat model structure. The rightmost part shows the binaural audio generation process.
  • Figure 2: Classification (mAP) and Direction-of-Arrival (MAE) for different $\lambda_1$ values.
  • Figure 3: Frequency distribution of training reverberations' distances ranging from 0 to 10 meters.
  • Figure 4: Sound events A and B occur in different directions and distances simultaneously.