Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment

Risa Shinoda, Kaede Shiohara, Nakamasa Inoue, Kuniaki Saito, Hiroaki Santo, Fumio Okura

Abstract

Understanding animal species from multimodal data poses an emerging challenge at the intersection of computer vision and ecology. While recent biological models, such as BioCLIP, have demonstrated strong alignment between images and textual taxonomic information for species identification, the integration of the audio modality remains an open problem. We propose BioVITA, a novel visual-textual-acoustic alignment framework for biological applications. BioVITA involves (i) a training dataset, (ii) a representation model, and (iii) a retrieval benchmark. First, we construct a large-scale training dataset comprising 1.3 million audio clips and 2.3 million images, covering 14,133 species annotated with 34 ecological trait labels. Second, building upon BioCLIP2, we introduce a two-stage training framework to effectively align audio representations with visual and textual representations. Third, we develop a cross-modal retrieval benchmark that covers all possible directional retrieval across the three modalities (i.e., image-to-audio, audio-to-text, text-to-image, and their reverse directions), with three taxonomic levels: Family, Genus, and Species. Extensive experiments demonstrate that our model learns a unified representation space that captures species-level semantics beyond taxonomy, advancing multimodal biodiversity understanding. The project page is available at: https://dahlian00.github.io/BioVITA_Page/

BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignment

Abstract

Understanding animal species from multimodal data poses an emerging challenge at the intersection of computer vision and ecology. While recent biological models, such as BioCLIP, have demonstrated strong alignment between images and textual taxonomic information for species identification, the integration of the audio modality remains an open problem. We propose BioVITA, a novel visual-textual-acoustic alignment framework for biological applications. BioVITA involves (i) a training dataset, (ii) a representation model, and (iii) a retrieval benchmark. First, we construct a large-scale training dataset comprising 1.3 million audio clips and 2.3 million images, covering 14,133 species annotated with 34 ecological trait labels. Second, building upon BioCLIP2, we introduce a two-stage training framework to effectively align audio representations with visual and textual representations. Third, we develop a cross-modal retrieval benchmark that covers all possible directional retrieval across the three modalities (i.e., image-to-audio, audio-to-text, text-to-image, and their reverse directions), with three taxonomic levels: Family, Genus, and Species. Extensive experiments demonstrate that our model learns a unified representation space that captures species-level semantics beyond taxonomy, advancing multimodal biodiversity understanding. The project page is available at: https://dahlian00.github.io/BioVITA_Page/
Paper Structure (63 sections, 2 equations, 15 figures, 13 tables)

This paper contains 63 sections, 2 equations, 15 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Species Recognition from Images.
  4. Species Recognition from Audio.
  5. Multi-modal Recognition.
  6. Dataset Comparison.
  7. Training Dataset for BioVITA
  8. Dataset Construction
  9. 1) Audio Data Curation.
  10. 2) Fine-Grained Annotation.
  11. 3) Visual Data Consolidation.
  12. Statistics and Examples
  13. Taxonomy.
  14. Audio Duration.
  15. Image Size.
  16. ...and 48 more sections

Figures (15)

  • Figure 1: We introduce $\mathsf{BioVITA}$ for biological visual-textual-acoustic alignment. We curate a million-scale dataset, train a unified representation model, and develop a comprehensive species-level retrieval benchmark.
  • Figure 2: Representative taxonomic distribution of the dataset across the five animal classes. Numbers indicate audio recordings.
  • Figure 3: Audio duration.
  • Figure 4: Image size.
  • Figure 5: Examples from $\mathsf{BioVITA\space Train}$. Images and audio clips are shown with their corresponding scientific names.
  • ...and 10 more figures