Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Meta Information for Audio-based Zero-shot Bird Classification

Alexander Gebhard, Andreas Triantafyllopoulos, Teresa Bez, Lukas Christ, Alexander Kathan, Björn W. Schuller

TL;DR

This work tackles data scarcity in audio-based bird identification by applying zero-shot learning with external metadata. It compares three metadata sources—textual field-guide descriptions encoded via (S)BERT, AVONET functional traits, and BLH life-history traits—using AST audio embeddings mapped to class embeddings through a linear projection and a dot-product compatibility trained with a ranking hinge loss. The key finding is that combining AVONET and BLH yields the best performance (mean F1 = 0.233) across five disjoint test sets, while textual descriptions underperform, likely due to onomatopoeia not being captured by language models. The study highlights the value of carefully chosen metadata for zero-shot audio bird classification and suggests future work on language-model pretraining for onomatopoeia and incorporating image data to further boost performance.

Abstract

Advances in passive acoustic monitoring and machine learning have led to the procurement of vast datasets for computational bioacoustic research. Nevertheless, data scarcity is still an issue for rare and underrepresented species. This study investigates how meta-information can improve zero-shot audio classification, utilising bird species as an example case study due to the availability of rich and diverse meta-data. We investigate three different sources of metadata: textual bird sound descriptions encoded via (S)BERT, functional traits (AVONET), and bird life-history (BLH) characteristics. As audio features, we extract audio spectrogram transformer (AST) embeddings and project them to the dimension of the auxiliary information by adopting a single linear layer. Then, we employ the dot product as compatibility function and a standard zero-shot learning ranking hinge loss to determine the correct class. The best results are achieved by concatenating the AVONET and BLH features attaining a mean unweighted F1-score of .233 over five different test sets with 8 to 10 classes.

Exploring Meta Information for Audio-based Zero-shot Bird Classification

TL;DR

This work tackles data scarcity in audio-based bird identification by applying zero-shot learning with external metadata. It compares three metadata sources—textual field-guide descriptions encoded via (S)BERT, AVONET functional traits, and BLH life-history traits—using AST audio embeddings mapped to class embeddings through a linear projection and a dot-product compatibility trained with a ranking hinge loss. The key finding is that combining AVONET and BLH yields the best performance (mean F1 = 0.233) across five disjoint test sets, while textual descriptions underperform, likely due to onomatopoeia not being captured by language models. The study highlights the value of carefully chosen metadata for zero-shot audio bird classification and suggests future work on language-model pretraining for onomatopoeia and incorporating image data to further boost performance.

Abstract

Advances in passive acoustic monitoring and machine learning have led to the procurement of vast datasets for computational bioacoustic research. Nevertheless, data scarcity is still an issue for rare and underrepresented species. This study investigates how meta-information can improve zero-shot audio classification, utilising bird species as an example case study due to the availability of rich and diverse meta-data. We investigate three different sources of metadata: textual bird sound descriptions encoded via (S)BERT, functional traits (AVONET), and bird life-history (BLH) characteristics. As audio features, we extract audio spectrogram transformer (AST) embeddings and project them to the dimension of the auxiliary information by adopting a single linear layer. Then, we employ the dot product as compatibility function and a standard zero-shot learning ranking hinge loss to determine the correct class. The best results are achieved by concatenating the AVONET and BLH features attaining a mean unweighted F1-score of .233 over five different test sets with 8 to 10 classes.
Paper Structure (10 sections, 5 equations, 2 figures, 1 table)

This paper contains 10 sections, 5 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Dataset
  3. Methodology
  4. Features
  5. Zero-Shot Classification
  6. Experimental Setup
  7. Results
  8. Conclusion
  9. Acknowledgement
  10. References

Figures (2)

  • Figure 1: The utilised ZSL pipeline, based on Xie21-ZAC.
  • Figure 2: The pairwise cosine similarities between the $95$ bird species for the Bert embeddings (left) and the SBert embeddings (right) visualised as heat maps. The SBert embeddings show a stronger difference among the birds.