Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent Multi-view Learning for Robust Environmental Sound Representations

Sivan Ding, Julia Wilkins, Magdalena Fuentes, Juan Pablo Bello

TL;DR

The paper tackles annotation scarcity in environmental sound learning by introducing latent multi-view learning that jointly leverages reconstruction and contrastive objectives. It uses a metadata-driven two-view setup to split information into view-private $z_p$ and view-shared $z_S$ subspaces, with a decoder reconstructing latent inputs and a classifier setup evaluating downstream sensor and source tasks. Empirical results on the SONYC-UST-V2 dataset show that reconstruction plus a separation-based cosine objective yields notable gains over DAC baselines and traditional SSL methods, with evidence of disentanglement between source and sensor attributes. The work highlights the potential of combined SSL strategies to produce robust, transferable environmental sound representations and suggests avenues for extending to next-generation audio understanding models and out-of-distribution data.

Abstract

Self-supervised learning (SSL) approaches, such as contrastive and generative methods, have advanced environmental sound representation learning using unlabeled data. However, how these approaches can complement each other within a unified framework remains relatively underexplored. In this work, we propose a multi-view learning framework that integrates contrastive principles into a generative pipeline to capture sound source and device information. Our method encodes compressed audio latents into view-specific and view-common subspaces, guided by two self-supervised objectives: contrastive learning for targeted information flow between subspaces, and reconstruction for overall information preservation. We evaluate our method on an urban sound sensor network dataset for sound source and sensor classification, demonstrating improved downstream performance over traditional SSL techniques. Additionally, we investigate the model's potential to disentangle environmental sound attributes within the structured latent space under varied training configurations.

Latent Multi-view Learning for Robust Environmental Sound Representations

TL;DR

The paper tackles annotation scarcity in environmental sound learning by introducing latent multi-view learning that jointly leverages reconstruction and contrastive objectives. It uses a metadata-driven two-view setup to split information into view-private and view-shared subspaces, with a decoder reconstructing latent inputs and a classifier setup evaluating downstream sensor and source tasks. Empirical results on the SONYC-UST-V2 dataset show that reconstruction plus a separation-based cosine objective yields notable gains over DAC baselines and traditional SSL methods, with evidence of disentanglement between source and sensor attributes. The work highlights the potential of combined SSL strategies to produce robust, transferable environmental sound representations and suggests avenues for extending to next-generation audio understanding models and out-of-distribution data.

Abstract

Self-supervised learning (SSL) approaches, such as contrastive and generative methods, have advanced environmental sound representation learning using unlabeled data. However, how these approaches can complement each other within a unified framework remains relatively underexplored. In this work, we propose a multi-view learning framework that integrates contrastive principles into a generative pipeline to capture sound source and device information. Our method encodes compressed audio latents into view-specific and view-common subspaces, guided by two self-supervised objectives: contrastive learning for targeted information flow between subspaces, and reconstruction for overall information preservation. We evaluate our method on an urban sound sensor network dataset for sound source and sensor classification, demonstrating improved downstream performance over traditional SSL techniques. Additionally, we investigate the model's potential to disentangle environmental sound attributes within the structured latent space under varied training configurations.

Paper Structure

This paper contains 16 sections, 7 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Multi-view Representation Learning
  4. Self-supervised Training Strategies
  5. Experimental Design
  6. Dataset and Multi-View Data Pairing
  7. Audio Preprocessing
  8. Training Recipe
  9. Downstream Classification
  10. Evaluation Metrics
  11. Baseline Methods
  12. Results and Discussion
  13. Main Comparisons
  14. Disentanglement Investigation
  15. Conclusion and Future Work
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Our self-supervised multi-view framework for environmental sound representation learning. Learned latent subspaces are used for downstream sound source and recording sensor classification tasks.
  • Figure 2: Visualizing the effects of cosine distance and masking strategies on downstream tasks using the overall performance vs. the Directional Subspace Classification metric ($DSC\Delta$).