Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Spatially-Aware Language and Audio Embeddings

Bhavika Devnani, Skyler Seto, Zakaria Aldeneh, Alessandro Toso, Elena Menyaylenko, Barry-John Theobald, Jonathan Sheaffer, Miguel Sarabia

TL;DR

This work presents ELSA, a spatially aware-audio and text embedding model trained using multimodal contrastive learning that supports non-spatial audio, spatial audio, and open vocabulary text captions describing both the spatial and semantic components of sound.

Abstract

Humans can picture a sound scene given an imprecise natural language description. For example, it is easy to imagine an acoustic environment given a phrase like "the lion roar came from right behind me!". For a machine to have the same degree of comprehension, the machine must know what a lion is (semantic attribute), what the concept of "behind" is (spatial attribute) and how these pieces of linguistic information align with the semantic and spatial attributes of the sound (what a roar sounds like when its coming from behind). State-of-the-art audio foundation models which learn to map between audio scenes and natural textual descriptions, are trained on non-spatial audio and text pairs, and hence lack spatial awareness. In contrast, sound event localization and detection models are limited to recognizing sounds from a fixed number of classes, and they localize the source to absolute position (e.g., 0.2m) rather than a position described using natural language (e.g., "next to me"). To address these gaps, we present ELSA a spatially aware-audio and text embedding model trained using multimodal contrastive learning. ELSA supports non-spatial audio, spatial audio, and open vocabulary text captions describing both the spatial and semantic components of sound. To train ELSA: (a) we spatially augment the audio and captions of three open-source audio datasets totaling 4,738 hours of audio, and (b) we design an encoder to capture the semantics of non-spatial audio, and the semantics and spatial attributes of spatial audio using contrastive learning. ELSA is competitive with state-of-the-art for both semantic retrieval and 3D source localization. In particular, ELSA achieves +2.8% mean audio-to-text and text-to-audio R@1 above the baseline, and outperforms by -11.6° mean-absolute-error in 3D source localization over the baseline.

Learning Spatially-Aware Language and Audio Embeddings

TL;DR

This work presents ELSA, a spatially aware-audio and text embedding model trained using multimodal contrastive learning that supports non-spatial audio, spatial audio, and open vocabulary text captions describing both the spatial and semantic components of sound.

Abstract

Humans can picture a sound scene given an imprecise natural language description. For example, it is easy to imagine an acoustic environment given a phrase like "the lion roar came from right behind me!". For a machine to have the same degree of comprehension, the machine must know what a lion is (semantic attribute), what the concept of "behind" is (spatial attribute) and how these pieces of linguistic information align with the semantic and spatial attributes of the sound (what a roar sounds like when its coming from behind). State-of-the-art audio foundation models which learn to map between audio scenes and natural textual descriptions, are trained on non-spatial audio and text pairs, and hence lack spatial awareness. In contrast, sound event localization and detection models are limited to recognizing sounds from a fixed number of classes, and they localize the source to absolute position (e.g., 0.2m) rather than a position described using natural language (e.g., "next to me"). To address these gaps, we present ELSA a spatially aware-audio and text embedding model trained using multimodal contrastive learning. ELSA supports non-spatial audio, spatial audio, and open vocabulary text captions describing both the spatial and semantic components of sound. To train ELSA: (a) we spatially augment the audio and captions of three open-source audio datasets totaling 4,738 hours of audio, and (b) we design an encoder to capture the semantics of non-spatial audio, and the semantics and spatial attributes of spatial audio using contrastive learning. ELSA is competitive with state-of-the-art for both semantic retrieval and 3D source localization. In particular, ELSA achieves +2.8% mean audio-to-text and text-to-audio R@1 above the baseline, and outperforms by -11.6° mean-absolute-error in 3D source localization over the baseline.
Paper Structure (36 sections, 12 equations, 7 figures, 16 tables)

This paper contains 36 sections, 12 equations, 7 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Audio-language Approaches
  4. Audio-language Datasets
  5. Paired Spatial Audio and Text Datasets
  6. Spatial Audio Encoding: First Order Ambisonics
  7. Spatial Augmentation of the Audio and Captions
  8. Spatial Real-World Dataset
  9. ELSA Pretraining for Spatial Audio and Language
  10. Audio Input Features
  11. Audio and Text Encoders
  12. Pretraining Objectives
  13. Experiments, Results, and Discussion
  14. Training and Evaluation of ELSA
  15. Spatial Attributes Evaluation
  16. ...and 21 more sections

Figures (7)

  • Figure 1: Our pipeline for learning spatial-audio representations aligned with natural language.
  • Figure 2: UMAP projection of ELSA embeddings of the test splits of Spatial-Clotho and Spatial-AudioCaps. Filled markers are obtained from spatial audio, and hollow markers are obtained from spatial captions. The UMAP projection was fitted with the train splits of Spatial-Clotho and Spatial-Audio caps, and we made use of supervised dimension reduction to highlight the direction differences rather than the semantic differences in the embeddings.
  • Figure A.F.3: Full architecture diagram for ELSA. Filled blocks include trainable parameters.
  • Figure A.F.4: Architecture diagram for Spatial Attributes Branch. Filled blocks include trainable parameters. The AddCoords2D block is described in liu2018intriguing.
  • Figure A.F.5: Boxplots of absolute direction-of arrival errors predicted by 2-layer MLP. Figs. (a)--(e) show the Spatial Audiocaps and Spatial Clotho test sets errors by different categories. Fig. (f) shows the predictions of the test set of TUT Sounds 2018 by different semantic classes. For all figures, boxes represent the interquartile range, solid orange lines are the median, and dashed green lines are the mean.
  • ...and 2 more figures