Short-Segment Speaker Verification with Pre-trained Models and Multi-Resolution Encoder

TL;DR

The paper tackles short-segment speaker verification (SV) by addressing the temporal-resolution limitations of self-supervised pre-trained models (PTMs). It proposes a fusion framework that combines PTM features with log-mel filterbank (FBank) features and augments them with a Multi-Resolution Encoder (MRE) whose outputs are injected into the ECAPA-TDNN backbone through adapters. The method employs layer-wise PTM fusion, and a learned weighting scheme directs information flow across multi-scale representations to maximize performance on segments shorter than 2 seconds. Evaluations on VoxCeleb demonstrate consistent improvements over prior feature configurations, with the approach especially benefitting from WavLM, HuBERT, or wav2vec 2.0 PTMs. A detailed analysis of the learned weights shows that MRE and FBank features steer PTM representations toward higher-level information, validating the complementary role of fine-grained temporal features in SV.

Abstract

Speaker verification (SV) utilizing features obtained from models pre-trained via self-supervised learning has recently demonstrated impressive performances. However, these pre-trained models (PTMs) usually have a temporal resolution of 20 ms, which is lower than typical filterbank features. It may be problematic especially for short-segment SV with an input segment shorter than 2 s, in which we need to extract as much information as possible from the input with a limited length. Although there have been approaches to utilize multi-resolution features from the HuBERT models, the window shifts were 20, 40, and 100 ms when the sampling rate was 16 kHz and thus only lower resolution features were considered. In this study, we propose an SV system which utilizes PTM features along with filterbank features and those from the multi-resolution time domain encoder with window shifts of 1.56, 3.13, 6.25, and 12.5 ms. Experimental results on the VoxCeleb dataset with various input lengths showed consistent improvements over systems with various combinations of input features.

Figures (3)

• Figure 1: Block diagram of the proposed short-segment speaker verification system with a pre-trained model and a multi-resolution encoder.
• Figure 2: Four combinations of input features for speaker verification.
• Figure 3: Learned layer-wise weights for WavLM-Large features in the systems with different combinations of input features.