Audio-to-Image Bird Species Retrieval without Audio-Image Pairs via Text Distillation

TL;DR

The paper tackles bioacoustic cross-modal retrieval under limited paired data by distilling visually grounded semantics from a pretrained image-text model into a pretrained audio-text model. It formalizes a contrastive loss with a learnable projection $g: \mathbb{R}^{d_A} \to \mathbb{R}^{d_I}$ and trains the audio encoder $f_A$ to match BioCLIP-2 text embeddings, while BioCLIP-2 remains fixed. The resulting BioLingual-FT exhibits emergent audio--image alignment, enabling audio-to-image retrieval that surpasses zero-shot and text-mapping baselines and preserves audio discriminative power, as shown on focal and soundscape benchmarks. This approach demonstrates that indirect semantic transfer through text is sufficient to achieve visually grounded cross-modal reasoning in data-scarce bioacoustic settings, offering a practical, lightweight solution for visually informed species recognition.

Abstract

Audio-to-image retrieval offers an interpretable alternative to audio-only classification for bioacoustic species recognition, but learning aligned audio-image representations is challenging due to the scarcity of paired audio-image data. We propose a simple and data-efficient approach that enables audio-to-image retrieval without any audio-image supervision. Our proposed method uses text as a semantic intermediary: we distill the text embedding space of a pretrained image-text model (BioCLIP-2), which encodes rich visual and taxonomic structure, into a pretrained audio-text model (BioLingual) by fine-tuning its audio encoder with a contrastive objective. This distillation transfers visually grounded semantics into the audio representation, inducing emergent alignment between audio and image embeddings without using images during training. We evaluate the resulting model on multiple bioacoustic benchmarks. The distilled audio encoder preserves audio discriminative power while substantially improving audio-text alignment on focal recordings and soundscape datasets. Most importantly, on the SSW60 benchmark, the proposed approach achieves strong audio-to-image retrieval performance exceeding baselines based on zero-shot model combinations or learned mappings between text embeddings, despite not training on paired audio-image data. These results demonstrate that indirect semantic transfer through text is sufficient to induce meaningful audio-image alignment, providing a practical solution for visually grounded species recognition in data-scarce bioacoustic settings.

Figures (2)

• Figure 1: Comparison of audio–image alignment approaches. Left: direct contrastive alignment using paired audio–image data. Right (ours): text-based distillation from BioCLIP-2 into the BioLingual audio encoder, inducing emergent audio–image alignment without audio–image training pairs.
• Figure 2: Audio-to-image retrieval: Images are embedded using BioCLIP-2 image encoder, while the audio query is embedded using BioLingual-FT audio encoder. Retrieval is performed by ranking images via cosine similarity in the shared embedding space.