Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Leveraging Pre-Trained Multi-Layer Representations for Speaker Verification

Jin Sob Kim, Hyun Joon Park, Wooseok Shin, Sung Won Han

TL;DR

Rethinking layer aggregation for speaker verification, this work introduces Layer Attentive Pooling (LAP), a time-dynamic inter-layer weighting mechanism, and pairs it with a lightweight backend using Attentive Statistical Temporal Pooling (ASTP). LAP leverages a multi-head projection and squeeze-and-excitation style weighting to assign layer importance and then applies max-pooling over layers, while ASTP captures temporal dynamics through channel- and context-dependent pooling to form embeddings. On VoxCeleb, the LAP+ASTP system achieves state-of-the-art or highly competitive results with substantially faster training times than heavier backends, demonstrating the practicality of fully exploiting pre-trained representations without extensive fine-tuning. These results highlight the value of time-varying, layer-aware aggregation for rich speaker cues in scalable SV pipelines.

Abstract

Recent speaker verification studies have achieved notable success by leveraging layer-wise output from pre-trained Transformer models. However, few have explored the advancements in aggregating these multi-level features beyond the static weighted average. We present Layer Attentive Pooling (LAP), a novel strategy for aggregating inter-layer representations from pre-trained speech models for speaker verification. LAP assesses the significance of each layer from multiple perspectives time-dynamically, and employs max pooling instead of averaging. Additionally, we propose a lightweight backend speaker model comprising LAP and Attentive Statistical Temporal Pooling (ASTP) to extract speaker embeddings from pre-trained model output. Experiments on the VoxCeleb benchmark reveal that our compact architecture achieves state-of-the-art performance while greatly reducing the training time. We further analyzed LAP design and its dynamic weighting mechanism for capturing speaker characteristics.

Rethinking Leveraging Pre-Trained Multi-Layer Representations for Speaker Verification

TL;DR

Rethinking layer aggregation for speaker verification, this work introduces Layer Attentive Pooling (LAP), a time-dynamic inter-layer weighting mechanism, and pairs it with a lightweight backend using Attentive Statistical Temporal Pooling (ASTP). LAP leverages a multi-head projection and squeeze-and-excitation style weighting to assign layer importance and then applies max-pooling over layers, while ASTP captures temporal dynamics through channel- and context-dependent pooling to form embeddings. On VoxCeleb, the LAP+ASTP system achieves state-of-the-art or highly competitive results with substantially faster training times than heavier backends, demonstrating the practicality of fully exploiting pre-trained representations without extensive fine-tuning. These results highlight the value of time-varying, layer-aware aggregation for rich speaker cues in scalable SV pipelines.

Abstract

Recent speaker verification studies have achieved notable success by leveraging layer-wise output from pre-trained Transformer models. However, few have explored the advancements in aggregating these multi-level features beyond the static weighted average. We present Layer Attentive Pooling (LAP), a novel strategy for aggregating inter-layer representations from pre-trained speech models for speaker verification. LAP assesses the significance of each layer from multiple perspectives time-dynamically, and employs max pooling instead of averaging. Additionally, we propose a lightweight backend speaker model comprising LAP and Attentive Statistical Temporal Pooling (ASTP) to extract speaker embeddings from pre-trained model output. Experiments on the VoxCeleb benchmark reveal that our compact architecture achieves state-of-the-art performance while greatly reducing the training time. We further analyzed LAP design and its dynamic weighting mechanism for capturing speaker characteristics.
Paper Structure (12 sections, 6 equations, 3 figures, 2 tables)

This paper contains 12 sections, 6 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Extracting Speaker Embedding with Pre-Trained Models
  3. Pre-Trained Models
  4. Layer Attentive Pooling
  5. Attentive Statistic Temporal Pooling
  6. Experimental Setup
  7. Results
  8. VoxCeleb benchmark
  9. Training Efficiency
  10. Analysis
  11. Conclusion
  12. Acknowledgements

Figures (3)

  • Figure 1: Approaches for leveraging latent features from speech pre-trained networks in speaker verification.
  • Figure 2: Comparing EER performance and training efficiency.
  • Figure 3: Comparison between the layer aggregation strategies.