Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsupervised Composable Representations for Audio

Giovanni Bindi, Philippe Esling

TL;DR

The paper tackles the challenge of learning compositional representations for audio in an unsupervised setting. It proposes a diffusion-based encoder–decoder framework that decomposes mixtures into latent components in a pre-trained EnCodec latent space, using a shared diffusion model and a fixed composition operator, and introduces a latent-space prior trained via masking to enable recomposition. Empirically, the approach achieves competitive unsupervised latent source separation on Slakh2100 and, in some cases, surpasses supervised baselines such as Demucs on SI-SIR, while also enabling unconditional generation and controllable variation with strong quality and diversity metrics. Overall, the work provides a scalable, lower-cost pathway to unsupervised compositional audio modeling with practical implications for source separation and generative audio tasks.

Abstract

Current generative models are able to generate high-quality artefacts but have been shown to struggle with compositional reasoning, which can be defined as the ability to generate complex structures from simpler elements. In this paper, we focus on the problem of compositional representation learning for music data, specifically targeting the fully-unsupervised setting. We propose a simple and extensible framework that leverages an explicit compositional inductive bias, defined by a flexible auto-encoding objective that can leverage any of the current state-of-art generative models. We demonstrate that our framework, used with diffusion models, naturally addresses the task of unsupervised audio source separation, showing that our model is able to perform high-quality separation. Our findings reveal that our proposal achieves comparable or superior performance with respect to other blind source separation methods and, furthermore, it even surpasses current state-of-art supervised baselines on signal-to-interference ratio metrics. Additionally, by learning an a-posteriori masking diffusion model in the space of composable representations, we achieve a system capable of seamlessly performing unsupervised source separation, unconditional generation, and variation generation. Finally, as our proposal works in the latent space of pre-trained neural audio codecs, it also provides a lower computational cost with respect to other neural baselines.

Unsupervised Composable Representations for Audio

TL;DR

The paper tackles the challenge of learning compositional representations for audio in an unsupervised setting. It proposes a diffusion-based encoder–decoder framework that decomposes mixtures into latent components in a pre-trained EnCodec latent space, using a shared diffusion model and a fixed composition operator, and introduces a latent-space prior trained via masking to enable recomposition. Empirically, the approach achieves competitive unsupervised latent source separation on Slakh2100 and, in some cases, surpasses supervised baselines such as Demucs on SI-SIR, while also enabling unconditional generation and controllable variation with strong quality and diversity metrics. Overall, the work provides a scalable, lower-cost pathway to unsupervised compositional audio modeling with practical implications for source separation and generative audio tasks.

Abstract

Current generative models are able to generate high-quality artefacts but have been shown to struggle with compositional reasoning, which can be defined as the ability to generate complex structures from simpler elements. In this paper, we focus on the problem of compositional representation learning for music data, specifically targeting the fully-unsupervised setting. We propose a simple and extensible framework that leverages an explicit compositional inductive bias, defined by a flexible auto-encoding objective that can leverage any of the current state-of-art generative models. We demonstrate that our framework, used with diffusion models, naturally addresses the task of unsupervised audio source separation, showing that our model is able to perform high-quality separation. Our findings reveal that our proposal achieves comparable or superior performance with respect to other blind source separation methods and, furthermore, it even surpasses current state-of-art supervised baselines on signal-to-interference ratio metrics. Additionally, by learning an a-posteriori masking diffusion model in the space of composable representations, we achieve a system capable of seamlessly performing unsupervised source separation, unconditional generation, and variation generation. Finally, as our proposal works in the latent space of pre-trained neural audio codecs, it also provides a lower computational cost with respect to other neural baselines.
Paper Structure (11 sections, 8 equations, 2 figures, 5 tables, 1 algorithm)

This paper contains 11 sections, 8 equations, 2 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Diffusion models
  4. Proposed approach
  5. Decomposition
  6. Recomposition
  7. Experiments and Results
  8. Decomposition
  9. Recomposition
  10. Discussion and Further Works
  11. Conclusions

Figures (2)

  • Figure 1: The overall architecture of our decomposition model. We first mix the sources, map the data $\mathbf{x}$ to the latent space through a frozen, pre-trained EnCodec model, and then decompose it into a set of latent variables (two shown here). These variables then condition a parameter-sharing diffusion model whose generation are then recomposed by an operator $C$.
  • Figure :