Can you Remove the Downstream Model for Speaker Recognition with Self-Supervised Speech Features?

TL;DR

The paper addresses the problem that downstream speaker verification models are typically designed for filter-bank inputs, potentially overlooking the speaker-information already present in self-supervised speech features. By freezing SSL models as general-purpose feature extractors and re-designing the downstream head, the authors show that a drastically simplified architecture can achieve equal or better performance, including a 97.51% reduction in parameters and a 29.93% average improvement on the SUPERB benchmark, with improved data efficiency (60% of the training data). They validate this through zero-shot analyses across multiple SSL models and ablations of the downstream components, demonstrating that SSL features carry robust speaker information and that some traditional components (e.g., frame-level encoders) may be unnecessary. The findings suggest practical benefits for deploying data-efficient, cost-effective speaker verification systems using frozen SSL representations and guide future downstream design choices when leveraging SSL features.

Abstract

Self-supervised features are typically used in place of filter-bank features in speaker verification models. However, these models were originally designed to ingest filter-bank features as inputs, and thus, training them on top of self-supervised features assumes that both feature types require the same amount of learning for the task. In this work, we observe that pre-trained self-supervised speech features inherently include information required for downstream speaker verification task, and therefore, we can simplify the downstream model without sacrificing performance. To this end, we revisit the design of the downstream model for speaker verification using self-supervised features. We show that we can simplify the model to use 97.51% fewer parameters while achieving a 29.93% average improvement in performance on SUPERB. Consequently, we show that the simplified downstream model is more data efficient compared to baseline--it achieves better performance with only 60% of the training data.

Figures (1)

• Figure 1: The simplified downstream model (D.3 from Table \ref{['tab:downstream_ablation']}) performs better with less data compared to full downstream model (D.0 from Table \ref{['tab:downstream_ablation']}). The equal error rate (EER, $\%$) on VoxCeleb1 is reported under three data conditions: $20\%$, $60\%$, and $100\%$ of the training speakers.