Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CA-MHFA: A Context-Aware Multi-Head Factorized Attentive Pooling for SSL-Based Speaker Verification

Junyi Peng, Ladislav Mošner, Lin Zhang, Oldřich Plchot, Themos Stafylakis, Lukáš Burget, Jan Černocký

TL;DR

This work tackles the limited ability of SSL-based speaker verification back-ends to leverage local temporal context. It introduces CA-MHFA, a context-aware, multi-head factorized attentive pooling mechanism that uses grouped, trainable queries with shared keys/values to model surrounding frames efficiently. Empirical results on VoxCeleb demonstrate state-of-the-art or competitive EERs with fewer parameters and faster convergence, and the approach generalizes to emotion recognition and anti-spoofing under SUPERB-style evaluation across multiple SSL models. Overall, CA-MHFA offers a versatile, resource-efficient back-end that enhances contextual representations for SSL-based speech classification tasks.

Abstract

Self-supervised learning (SSL) models for speaker verification (SV) have gained significant attention in recent years. However, existing SSL-based SV systems often struggle to capture local temporal dependencies and generalize across different tasks. In this paper, we propose context-aware multi-head factorized attentive pooling (CA-MHFA), a lightweight framework that incorporates contextual information from surrounding frames. CA-MHFA leverages grouped, learnable queries to effectively model contextual dependencies while maintaining efficiency by sharing keys and values across groups. Experimental results on the VoxCeleb dataset show that CA-MHFA achieves EERs of 0.42\%, 0.48\%, and 0.96\% on Vox1-O, Vox1-E, and Vox1-H, respectively, outperforming complex models like WavLM-TDNN with fewer parameters and faster convergence. Additionally, CA-MHFA demonstrates strong generalization across multiple SSL models and tasks, including emotion recognition and anti-spoofing, highlighting its robustness and versatility.

CA-MHFA: A Context-Aware Multi-Head Factorized Attentive Pooling for SSL-Based Speaker Verification

TL;DR

This work tackles the limited ability of SSL-based speaker verification back-ends to leverage local temporal context. It introduces CA-MHFA, a context-aware, multi-head factorized attentive pooling mechanism that uses grouped, trainable queries with shared keys/values to model surrounding frames efficiently. Empirical results on VoxCeleb demonstrate state-of-the-art or competitive EERs with fewer parameters and faster convergence, and the approach generalizes to emotion recognition and anti-spoofing under SUPERB-style evaluation across multiple SSL models. Overall, CA-MHFA offers a versatile, resource-efficient back-end that enhances contextual representations for SSL-based speech classification tasks.

Abstract

Self-supervised learning (SSL) models for speaker verification (SV) have gained significant attention in recent years. However, existing SSL-based SV systems often struggle to capture local temporal dependencies and generalize across different tasks. In this paper, we propose context-aware multi-head factorized attentive pooling (CA-MHFA), a lightweight framework that incorporates contextual information from surrounding frames. CA-MHFA leverages grouped, learnable queries to effectively model contextual dependencies while maintaining efficiency by sharing keys and values across groups. Experimental results on the VoxCeleb dataset show that CA-MHFA achieves EERs of 0.42\%, 0.48\%, and 0.96\% on Vox1-O, Vox1-E, and Vox1-H, respectively, outperforming complex models like WavLM-TDNN with fewer parameters and faster convergence. Additionally, CA-MHFA demonstrates strong generalization across multiple SSL models and tasks, including emotion recognition and anti-spoofing, highlighting its robustness and versatility.
Paper Structure (16 sections, 3 equations, 1 figure, 3 tables)

This paper contains 16 sections, 3 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Context-aware Multi-head Factorized Attentive Pooling
  3. Frame-Level Extractor with Compression
  4. Context-Aware Attentive Pooling
  5. Utterance-Level Extractor
  6. Generalization of Other Works
  7. Experiments
  8. Setup
  9. Two Types of Evaluation
  10. Datasets
  11. Implementation details
  12. Performance Metrics
  13. Hyper-Parameter Analysis of CA-MHFA
  14. Comparison with State-of-the-art SV Systems
  15. CA-MHFA on Other Tasks
  16. ...and 1 more sections

Figures (1)

  • Figure 1: The architecture of the pre-trained model and attached lightweight speaker extractor back-end (context-aware multi-head factorized attentive pooling, CA-MHFA). During the fine-tuning, the SSL model and cascaded speaker extractor are jointly optimized.