Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Auden-Voice: General-Purpose Voice Encoder for Speech and Language Understanding

Mingyue Huo, Wei-Cheng Tseng, Yiwen Shao, Hao Zhang, Dong Yu

TL;DR

The paper addresses the bottleneck in large audio–language models where acoustic encoders struggle to balance speaker identity with paralinguistic cues. It systematically evaluates initialization strategies and training regimes, comparing multi-task learning to CLAP-style speech–text alignment on a Zipformer-based encoder, and assesses integration with LLMs via LLM-QA. The findings show that ASR-based initialization combined with multi-task supervision yields the most balanced representations across identity and paralinguistic tasks, while CLAP improves retrieval but can degrade other capabilities; LLM-QA grounding correlates with linear probing, with the multi-task encoder delivering strongest performance. The result is Auden-voice, a general-purpose, LLM-friendly voice encoder that balances cues and can be integrated into the Auden audio understanding toolkit for practical speech understanding and reasoning tasks.

Abstract

Human voice encodes both identity and paralinguistic cues, yet encoders in large audio-language models (LALMs) rarely balance both aspects. In this work, we present a study toward building a general-purpose voice encoder that captures nuanced voice cues. Through a comprehensive evaluation, we find that multi-task training yields the most balanced representations, whereas contrastive language-audio pretraining (CLAP) primarily improves retrieval without enhancing paralinguistic understanding. Our final encoder, Auden-Voice, also demonstrates strong performance when integrated with LLMs. The code and training recipes will be released with the audio understanding toolkit Auden.

Auden-Voice: General-Purpose Voice Encoder for Speech and Language Understanding

TL;DR

The paper addresses the bottleneck in large audio–language models where acoustic encoders struggle to balance speaker identity with paralinguistic cues. It systematically evaluates initialization strategies and training regimes, comparing multi-task learning to CLAP-style speech–text alignment on a Zipformer-based encoder, and assesses integration with LLMs via LLM-QA. The findings show that ASR-based initialization combined with multi-task supervision yields the most balanced representations across identity and paralinguistic tasks, while CLAP improves retrieval but can degrade other capabilities; LLM-QA grounding correlates with linear probing, with the multi-task encoder delivering strongest performance. The result is Auden-voice, a general-purpose, LLM-friendly voice encoder that balances cues and can be integrated into the Auden audio understanding toolkit for practical speech understanding and reasoning tasks.

Abstract

Human voice encodes both identity and paralinguistic cues, yet encoders in large audio-language models (LALMs) rarely balance both aspects. In this work, we present a study toward building a general-purpose voice encoder that captures nuanced voice cues. Through a comprehensive evaluation, we find that multi-task training yields the most balanced representations, whereas contrastive language-audio pretraining (CLAP) primarily improves retrieval without enhancing paralinguistic understanding. Our final encoder, Auden-Voice, also demonstrates strong performance when integrated with LLMs. The code and training recipes will be released with the audio understanding toolkit Auden.

Paper Structure

This paper contains 10 sections, 2 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Encoder Model
  4. Training Strategies
  5. Evaluation Setup
  6. Results
  7. Insights from Initialization and Supervision
  8. Multi-task vs. CLAP for Balanced Representation
  9. LLM-QA with Frozen Voice Encoder
  10. Conclusion

Figures (2)

  • Figure 1: Left: Zipformer encoder used across all settings to ensure fair evaluation. Right: Five evaluation setups covering both speech and speech-language understanding. Encoder is frozen; all heads are minimal to isolate voice representation quality.
  • Figure 2: Task-specific effects of multi-task training and CLAP fine-tuning, comparing unbalanced baseline, multi-task, CLAP-tuned, and multi-task+CLAP encoders.