Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WavLink: Compact Audio-Text Embeddings with a Global Whisper Token

Gokul Karthik Kumar, Ludovick Lepauloux, Hakim Hacid

TL;DR

WavLink addresses the gap between Whisper-based audio representations used in audio-LLMs and compact audio–text embeddings. By inserting a learnable global token into Whisper and jointly training with a text encoder under CLIP or SigLIP objectives, along with a two-stage training regime and Matryoshka supervision, it achieves state-of-the-art retrieval on AudioCaps and Clotho while producing sub-100D embeddings. The approach also delivers competitive zero-shot performance on VGGSound and strong results on AIR-Bench MCQs, with substantial gains in storage and search efficiency due to the single-token, scalable embeddings. These findings highlight the versatility of Whisper beyond ASR and point to practical benefits for large-scale, cross-modal retrieval and multi-task audio understanding, with potential extensions to multilingual alignment and audio–LLM integration.

Abstract

Whisper has become the de-facto encoder for extracting general-purpose audio features in large audio-language models, where a 30-second clip is typically represented by 1500 frame features projected into an LLM. In contrast, audio-text embedding models like CLAP-based models have largely relied on alternative audio encoders (e.g., HTS-AT, PaSST), and have not leveraged Whisper effectively. We present WavLink, a compact audio-text embedding model that augments Whisper encoder with a learnable global token, trained jointly with a text encoder. Through a systematic study of design choices, including pretrained text encoders, loss functions, training modes, and data mixtures, we identify configurations that yield state-of-the-art retrieval performance. Our two-stage training recipe across three model sizes, combined with Matryoshka-style supervision, improves scalability, enabling 8x smaller embeddings with minimal performance drop. WavLink also demonstrates competitive performance on AIR-Bench with MCQs and zero-shot classification.

WavLink: Compact Audio-Text Embeddings with a Global Whisper Token

TL;DR

WavLink addresses the gap between Whisper-based audio representations used in audio-LLMs and compact audio–text embeddings. By inserting a learnable global token into Whisper and jointly training with a text encoder under CLIP or SigLIP objectives, along with a two-stage training regime and Matryoshka supervision, it achieves state-of-the-art retrieval on AudioCaps and Clotho while producing sub-100D embeddings. The approach also delivers competitive zero-shot performance on VGGSound and strong results on AIR-Bench MCQs, with substantial gains in storage and search efficiency due to the single-token, scalable embeddings. These findings highlight the versatility of Whisper beyond ASR and point to practical benefits for large-scale, cross-modal retrieval and multi-task audio understanding, with potential extensions to multilingual alignment and audio–LLM integration.

Abstract

Whisper has become the de-facto encoder for extracting general-purpose audio features in large audio-language models, where a 30-second clip is typically represented by 1500 frame features projected into an LLM. In contrast, audio-text embedding models like CLAP-based models have largely relied on alternative audio encoders (e.g., HTS-AT, PaSST), and have not leveraged Whisper effectively. We present WavLink, a compact audio-text embedding model that augments Whisper encoder with a learnable global token, trained jointly with a text encoder. Through a systematic study of design choices, including pretrained text encoders, loss functions, training modes, and data mixtures, we identify configurations that yield state-of-the-art retrieval performance. Our two-stage training recipe across three model sizes, combined with Matryoshka-style supervision, improves scalability, enabling 8x smaller embeddings with minimal performance drop. WavLink also demonstrates competitive performance on AIR-Bench with MCQs and zero-shot classification.
Paper Structure (16 sections, 5 equations, 1 figure, 4 tables)

This paper contains 16 sections, 5 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Model Architecture
  4. Training Objectives
  5. Training Strategies
  6. Design Sweep Configuration
  7. Matryoshka Loss Adaptation
  8. Experiments
  9. Experimental Setup
  10. Design Sweep Results
  11. Retrieval Performance
  12. Generalization Beyond Retrieval: ZSC
  13. Generalization Beyond Retrieval: MCQ
  14. Scalability
  15. Ablations
  16. ...and 1 more sections

Figures (1)

  • Figure 1: Design sweep based on Recall@1 retrieval performance on the AudioCaps and Clotho benchmarks, identifying a fully trained CLIP-BERT model with CLIP's contrastive loss as the optimal configuration.