Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Compositional Structure in Audio Representations

Chuyang Chen, Bea Steers, Brian McFee, Juan Bello

Abstract

We propose a benchmark for evaluating compositionality in audio representations. Audio compositionality refers to representing sound scenes in terms of constituent sources and attributes, and combining them systematically. While central to auditory perception, this property is largely absent from current evaluation protocols. Our framework adapts ideas from vision and language to audio through two tasks: A-COAT, which tests consistency under additive transformations, and A-TRE, which probes reconstructibility from attribute-level primitives. Both tasks are supported by large synthetic datasets with controlled variation in acoustic attributes, providing the first benchmark of compositional structure in audio embeddings.

Evaluating Compositional Structure in Audio Representations

Abstract

We propose a benchmark for evaluating compositionality in audio representations. Audio compositionality refers to representing sound scenes in terms of constituent sources and attributes, and combining them systematically. While central to auditory perception, this property is largely absent from current evaluation protocols. Our framework adapts ideas from vision and language to audio through two tasks: A-COAT, which tests consistency under additive transformations, and A-TRE, which probes reconstructibility from attribute-level primitives. Both tasks are supported by large synthetic datasets with controlled variation in acoustic attributes, providing the first benchmark of compositional structure in audio embeddings.
Paper Structure (18 sections, 5 equations, 2 figures, 1 table)

This paper contains 18 sections, 5 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluating Audio Representations
  4. Evaluating Compositionality
  5. Method
  6. Audio Scenes
  7. A-COAT
  8. A-TRE
  9. Experiment
  10. Dataset Generation
  11. Data Balancing
  12. Model Selection
  13. Evaluation Protocol
  14. Results and Discussion
  15. Overall Results
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Model score distributions for A-COAT (a) and A-TRE (b) as notched box plots. Boxes show the interquartile range with median in red; notches give an approximate 95% confidence interval.
  • Figure 2: Model scores as a function of diversity. (a) A-COAT vs $H^{\mathrm{quad}}$: most models exhibit consistent negative slopes except BEATs. (b) A-TRE vs $H$: slopes vary across models, indicating differing sensitivity to diversity. Red lines show linear fits with 95% confidence intervals.