Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bird Vocalization Embedding Extraction Using Self-Supervised Disentangled Representation Learning

Runwu Shi, Katsutoshi Itoyama, Kazuhiro Nakadai

TL;DR

The paper tackles song-level embedding of bird vocalizations by introducing a disentangled representation learning framework with two encoders: a global encoder for temporal structure and a local encoder for discriminative content. The latent spaces $Z_G$ and $Z_L$ are learned under a VAE-like objective with capacity constraints, encouraging disentanglement and informativeness, and are decoded to reconstruct the input Mel-spectrogram. On Great Tit data, the local-encoder embeddings achieve high clustering quality (NMI ~0.901–0.902), outperforming vanilla VAE and several pretrained baselines, with effective compression to a smaller set of informative units (27 units) without loss. This approach enables scalable analysis of large vocal repertoires and provides insight into the distribution of information across embedding units for bioacoustic tasks.

Abstract

This paper addresses the extraction of the bird vocalization embedding from the whole song level using disentangled representation learning (DRL). Bird vocalization embeddings are necessary for large-scale bioacoustic tasks, and self-supervised methods such as Variational Autoencoder (VAE) have shown their performance in extracting such low-dimensional embeddings from vocalization segments on the note or syllable level. To extend the processing level to the entire song instead of cutting into segments, this paper regards each vocalization as the generalized and discriminative part and uses two encoders to learn these two parts. The proposed method is evaluated on the Great Tits dataset according to the clustering performance, and the results outperform the compared pre-trained models and vanilla VAE. Finally, this paper analyzes the informative part of the embedding, further compresses its dimension, and explains the disentangled performance of bird vocalizations.

Bird Vocalization Embedding Extraction Using Self-Supervised Disentangled Representation Learning

TL;DR

The paper tackles song-level embedding of bird vocalizations by introducing a disentangled representation learning framework with two encoders: a global encoder for temporal structure and a local encoder for discriminative content. The latent spaces and are learned under a VAE-like objective with capacity constraints, encouraging disentanglement and informativeness, and are decoded to reconstruct the input Mel-spectrogram. On Great Tit data, the local-encoder embeddings achieve high clustering quality (NMI ~0.901–0.902), outperforming vanilla VAE and several pretrained baselines, with effective compression to a smaller set of informative units (27 units) without loss. This approach enables scalable analysis of large vocal repertoires and provides insight into the distribution of information across embedding units for bioacoustic tasks.

Abstract

This paper addresses the extraction of the bird vocalization embedding from the whole song level using disentangled representation learning (DRL). Bird vocalization embeddings are necessary for large-scale bioacoustic tasks, and self-supervised methods such as Variational Autoencoder (VAE) have shown their performance in extracting such low-dimensional embeddings from vocalization segments on the note or syllable level. To extend the processing level to the entire song instead of cutting into segments, this paper regards each vocalization as the generalized and discriminative part and uses two encoders to learn these two parts. The proposed method is evaluated on the Great Tits dataset according to the clustering performance, and the results outperform the compared pre-trained models and vanilla VAE. Finally, this paper analyzes the informative part of the embedding, further compresses its dimension, and explains the disentangled performance of bird vocalizations.
Paper Structure (10 sections, 3 equations, 4 figures, 1 table)

This paper contains 10 sections, 3 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Proposed method
  3. Structure
  4. Training strategy
  5. Experiment
  6. Dataset
  7. Experiment and Results
  8. Analysis and Discussion
  9. Conclusions
  10. Acknowledgements

Figures (4)

  • Figure 1: Different elements in bird song (Great Tit).
  • Figure 2: The framework of the proposed method.
  • Figure 3: T-SNE of embeddings of compared methods with different colors for different song types. (a) Local encoder of the proposed method, (b) VAE (Global Encoder with Decoder), (c) Wav2Vec2, (d) Hubert, (e) VQ-APC, (f) OpenL3.
  • Figure 4: (a) Each unit's element-wise mean and variance, (b) KL divergence of each unit in embedding, (c) Reconstruction results using all or only the global encoder, (d) Reconstruction results when changing informative units in embedding.