Data-Efficient Self-Supervised Algorithms for Fine-Grained Birdsong Analysis

TL;DR

This work presents a lightweight, yet performant neural network architecture for birdsong annotation called Residual-MLP-RNN, which presents a robust three-stage training pipeline for developing reliable deep birdsong syllable detectors with minimal expert labor.

Abstract

Many bioacoustics, neuroscience, and linguistics research utilize birdsongs as proxy models to acquire knowledge in diverse areas. Developing models generally requires precisely annotated data at the level of syllables. Hence, automated and data-efficient methods that reduce annotation costs are in demand. This work presents a lightweight, yet performant neural network architecture for birdsong annotation called Residual-MLP-RNN. Then, it presents a robust three-stage training pipeline for developing reliable deep birdsong syllable detectors with minimal expert labor. The first stage is self-supervised learning from unlabeled data. Two of the most successful pretraining paradigms are explored, namely, masked prediction and online clustering. The second stage is supervised training with effective data augmentations to create a robust model for frame-level syllable detection. The third stage is semi-supervised post-training, which leverages the unlabeled data again. However, unlike the initial phase, this time it is aligned with the downstream task. The performance of this data-efficient approach is demonstrated for the complex song of the Canary in extreme label-scarcity scenarios. Canary has one of the most difficult songs to annotate, which implicitly validates the method for other birds. Finally, the potential of self-supervised embeddings is assessed for linear probing and unsupervised birdsong analysis.

Figures (7)

• Figure 1: An example of syllable prediction using the proposed model and the three-stage training framework.
• Figure 2: The proposed Res-MLP-RNN neural network architecture with an input spectrogram example. The model has roughly 10 M parameters. The Masked Prediction and Online Clustering heads are used in two separate self-supervised pretraining tasks. The Classifier head is used for supervised and post-training semi-supervised syllable detection tasks. The first linear layer of the first block projects 256 frequency bins into a 512-dimensional feature space. The intermediate dimension is fixed at 512 in all layers, except for the bi-directional LSTM, where a linear projection reverts its output from 1024 to 512 dimensions. FC is Fully Connected or linear layer; LN is LayerNorm; Drop is Dropout; Prototypes are learnable cluster centers.
• Figure 3: An example from the birdsong MAE model for the masked prediction task.
• Figure 4: Online Syllable Clustering loss.
• Figure 5: Two syllables from each bird with their true and predicted distribution of duration across training sizes and models. MAE means the model was pretrained by the SSL masked prediction prior to the supervised finetuning. Equivalently, OSC refers to the Online Syllable Clustering pretraining task. The predictions are faithful, even with such small training sizes.