Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SoCov: Semi-Orthogonal Parametric Pooling of Covariance Matrix for Speaker Recognition

Rongjin Li, Weibin Zhang, Dongpeng Chen, Jintao Kang, Xiaofen Xing

TL;DR

The paper tackles the limitation of conventional mean–std pooling in speaker recognition by introducing SoCov, a semi-orthogonal parametric pooling of the self-attentive covariance matrix. SoCov compresses the covariance via a trainable vector under a semi-orthogonal constraint to produce a compact second-order statistic (the cov-vec), which is concatenated with the weighted standard deviation to form the sc-vector input for segment-level processing. Empirical results on SRE21 development and evaluation sets show that sc-vector outperforms the traditional x-vector baseline (mean+std), especially when paired with self-attentive features, achieving substantial relative reductions in EER (e.g., around 30% on SRE21Eval) and proving robust across backbones such as FTDNN, ResNet34, and ECAPA-TDNN. The approach offers a practical, GPU-friendly way to leverage covariance information in deep speaker embeddings, improving discriminability without prohibitive computational cost.

Abstract

In conventional deep speaker embedding frameworks, the pooling layer aggregates all frame-level features over time and computes their mean and standard deviation statistics as inputs to subsequent segment-level layers. Such statistics pooling strategy produces fixed-length representations from variable-length speech segments. However, this method treats different frame-level features equally and discards covariance information. In this paper, we propose the Semi-orthogonal parameter pooling of Covariance matrix (SoCov) method. The SoCov pooling computes the covariance matrix from the self-attentive frame-level features and compresses it into a vector using the semi-orthogonal parametric vectorization, which is then concatenated with the weighted standard deviation vector to form inputs to the segment-level layers. Deep embedding based on SoCov is called ``sc-vector''. The proposed sc-vector is compared to several different baselines on the SRE21 development and evaluation sets. The sc-vector system significantly outperforms the conventional x-vector system, with a relative reduction in EER of 15.5% on SRE21Eval. When using self-attentive deep feature, SoCov helps to reduce EER on SRE21Eval by about 30.9% relatively to the conventional ``mean + standard deviation'' statistics.

SoCov: Semi-Orthogonal Parametric Pooling of Covariance Matrix for Speaker Recognition

TL;DR

The paper tackles the limitation of conventional mean–std pooling in speaker recognition by introducing SoCov, a semi-orthogonal parametric pooling of the self-attentive covariance matrix. SoCov compresses the covariance via a trainable vector under a semi-orthogonal constraint to produce a compact second-order statistic (the cov-vec), which is concatenated with the weighted standard deviation to form the sc-vector input for segment-level processing. Empirical results on SRE21 development and evaluation sets show that sc-vector outperforms the traditional x-vector baseline (mean+std), especially when paired with self-attentive features, achieving substantial relative reductions in EER (e.g., around 30% on SRE21Eval) and proving robust across backbones such as FTDNN, ResNet34, and ECAPA-TDNN. The approach offers a practical, GPU-friendly way to leverage covariance information in deep speaker embeddings, improving discriminability without prohibitive computational cost.

Abstract

In conventional deep speaker embedding frameworks, the pooling layer aggregates all frame-level features over time and computes their mean and standard deviation statistics as inputs to subsequent segment-level layers. Such statistics pooling strategy produces fixed-length representations from variable-length speech segments. However, this method treats different frame-level features equally and discards covariance information. In this paper, we propose the Semi-orthogonal parameter pooling of Covariance matrix (SoCov) method. The SoCov pooling computes the covariance matrix from the self-attentive frame-level features and compresses it into a vector using the semi-orthogonal parametric vectorization, which is then concatenated with the weighted standard deviation vector to form inputs to the segment-level layers. Deep embedding based on SoCov is called ``sc-vector''. The proposed sc-vector is compared to several different baselines on the SRE21 development and evaluation sets. The sc-vector system significantly outperforms the conventional x-vector system, with a relative reduction in EER of 15.5% on SRE21Eval. When using self-attentive deep feature, SoCov helps to reduce EER on SRE21Eval by about 30.9% relatively to the conventional ``mean + standard deviation'' statistics.

Paper Structure

This paper contains 14 sections, 14 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Statistics pooling
  4. Self-Attentive pooling
  5. Proposed Methods
  6. Covariance Statistics
  7. Parametric Pooling
  8. Experimental configuration
  9. Training Setup
  10. Models Setup
  11. Experimental results
  12. Effectiveness of constraints
  13. Effectiveness on Other Backbones
  14. Conclusion

Figures (2)

  • Figure 1: Diagram of the semi-orthogonal parametric pooling of self-attentive covariance matrix. The standard deviation and covariance matrix are firstly computed by using the attentive frame-level deep features. The covariance matrix is then vectorized and concatenated with the standard deviation to form inputs for the segment-level network.
  • Figure 2: The DET curves of self-attentive pooling (SAP) FTDNN systems with different statistics on NIST SRE21Eval.