Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encoder

TL;DR

This work proposes a unified post-training framework that enables a single speech foundation model to generate multiple types of utterance-level representations and demonstrates the effectiveness of this approach by jointly learning semantic and speaker representations and evaluating them on multilingual speech retrieval and speaker recognition tasks.

Abstract

Speech foundation models trained with self-supervised learning produce generic speech representations that support a wide range of speech processing tasks. When further adapted with supervised learning, these models can achieve strong performance on specific downstream tasks. Recent post-training approaches, such as SAMU-XSLR and SONAR, align speech representations with utterance-level semantic representations, enabling effective multimodal (speech-text) and multilingual applications. While speech foundation models typically learn contextual embeddings at the acoustic frame level, these methods learn representations at the utterance level. In this work, we extend this paradigm to arbitrary utterance-level attributes and propose a unified post-training framework that enables a single speech foundation model to generate multiple types of utterance-level representations. We demonstrate the effectiveness of this approach by jointly learning semantic and speaker representations and evaluating them on multilingual speech retrieval and speaker recognition tasks.

Figures (2)

• Figure 1: Attribute-specific branch used to learn an utterance-level representation for attribute $\tau$ in the teacher–student framework. Layer representations from the shared SSL encoder are projected, combined using learnable interpolation weights $\lambda_{\tau,\ell}$, and aggregated by attention pooling to produce an embedding aligned with the teacher representation.
• Figure 2: Learned layer-interpolation weights for the semantic and speaker branches of the unified speech encoder.