Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Compositional Audio Representation Learning

Sripathi Sridhar, Mark Cartwright

TL;DR

This work addresses the challenge of learning interpretable, source-centric representations for complex auditory scenes, moving beyond clip-level encodings. It introduces Compositional Audio Representation Learning (CARL), with supervised and unsupervised variants built around a frozen AudioMAE encoder, a slot-transformer that yields multiple source slots, a reconstruction decoder, and a slot-wise classifier, trained with a mixture of reconstruction and regularization losses. On the OST synthetic dataset, supervised CARL achieves the best performance, and unsupervised CARL shows that feature reconstruction and larger slot dimensions improve open-ended source discovery, while reconstruction targets and regularizers crucially shape generalization. The results suggest that supervised signals enhance interpretability and downstream performance, while unsupervised approaches require richer signals to close the gap, pointing to future work with partial labels, counts, natural language descriptions, or self-supervision to enable robust, source-level machine listening in real-world environments.

Abstract

Human auditory perception is compositional in nature -- we identify auditory streams from auditory scenes with multiple sound events. However, such auditory scenes are typically represented using clip-level representations that do not disentangle the constituent sound sources. In this work, we learn source-centric audio representations where each sound source is represented using a distinct, disentangled source embedding in the audio representation. We propose two novel approaches to learning source-centric audio representations: a supervised model guided by classification and an unsupervised model guided by feature reconstruction, both of which outperform the baselines. We thoroughly evaluate the design choices of both approaches using an audio classification task. We find that supervision is beneficial to learn source-centric representations, and that reconstructing audio features is more useful than reconstructing spectrograms to learn unsupervised source-centric representations. Leveraging source-centric models can help unlock the potential of greater interpretability and more flexible decoding in machine listening.

Compositional Audio Representation Learning

TL;DR

This work addresses the challenge of learning interpretable, source-centric representations for complex auditory scenes, moving beyond clip-level encodings. It introduces Compositional Audio Representation Learning (CARL), with supervised and unsupervised variants built around a frozen AudioMAE encoder, a slot-transformer that yields multiple source slots, a reconstruction decoder, and a slot-wise classifier, trained with a mixture of reconstruction and regularization losses. On the OST synthetic dataset, supervised CARL achieves the best performance, and unsupervised CARL shows that feature reconstruction and larger slot dimensions improve open-ended source discovery, while reconstruction targets and regularizers crucially shape generalization. The results suggest that supervised signals enhance interpretability and downstream performance, while unsupervised approaches require richer signals to close the gap, pointing to future work with partial labels, counts, natural language descriptions, or self-supervision to enable robust, source-level machine listening in real-world environments.

Abstract

Human auditory perception is compositional in nature -- we identify auditory streams from auditory scenes with multiple sound events. However, such auditory scenes are typically represented using clip-level representations that do not disentangle the constituent sound sources. In this work, we learn source-centric audio representations where each sound source is represented using a distinct, disentangled source embedding in the audio representation. We propose two novel approaches to learning source-centric audio representations: a supervised model guided by classification and an unsupervised model guided by feature reconstruction, both of which outperform the baselines. We thoroughly evaluate the design choices of both approaches using an audio classification task. We find that supervision is beneficial to learn source-centric representations, and that reconstructing audio features is more useful than reconstructing spectrograms to learn unsupervised source-centric representations. Leveraging source-centric models can help unlock the potential of greater interpretability and more flexible decoding in machine listening.
Paper Structure (19 sections, 1 figure, 6 tables)

This paper contains 19 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Compositional Audio Representation Learning
  3. Audio encoder
  4. Slot transformer
  5. Reconstruction decoder
  6. Classifier
  7. Loss terms
  8. Evaluation
  9. Data
  10. Training
  11. Evaluation metrics
  12. Baselines
  13. Experiments
  14. Results
  15. Design choices for supervised CARL
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Proposed compositional audio representation learning (CARL) model consisting of a frozen audio encoder, slot transformer, reconstruction decoder, and classifier. The modules in yellow are trained jointly using a combined weighted loss function, including a reconstruction objective on the unsupervised branch, and a permutation-invariant cross-entropy loss on the supervised branch.