Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Speaker Embedding Self-Augmentation for Personal Voice Activity Detection with Short Enrollment Speech

Fuyuan Feng, Wenbin Zhang, Yu Gao, Longting Xu, Xiaofeng Mou, Yi Xu

TL;DR

This work tackles Personal Voice Activity Detection under short enrollment by introducing an adaptive speaker embedding self-augmentation framework and a long-term embedding adaptation strategy. The method augments enrollment embeddings with a frame-level embedding extracted from mixed speech and refines these embeddings over multiple segments using a weighted update rule, all without retraining. Empirical results show notable improvements in recall, precision, and F1-score for short enrollment, with the long-term approach achieving parity with full enrollment after five iterations. The approach offers practical benefits for PVAD in real-world, wake-word-driven scenarios by enhancing personalization with limited enrollment data.

Abstract

Personal Voice Activity Detection (PVAD) is crucial for identifying target speaker segments in the mixture, yet its performance heavily depends on the quality of speaker embeddings. A key practical limitation is the short enrollment speech--such as a wake-up word--which provides limited cues. This paper proposes a novel adaptive speaker embedding self-augmentation strategy that enhances PVAD performance by augmenting the original enrollment embeddings through additive fusion of keyframe embeddings extracted from mixed speech. Furthermore, we introduce a long-term adaptation strategy to iteratively refine embeddings during detection, mitigating speaker temporal variability. Experiments show significant gains in recall, precision, and F1-score under short enrollment conditions, matching full-length enrollment performance after five iterative updates. The source code is available at https://anonymous.4open.science/r/ASE-PVAD-E5D6 .

Adaptive Speaker Embedding Self-Augmentation for Personal Voice Activity Detection with Short Enrollment Speech

TL;DR

This work tackles Personal Voice Activity Detection under short enrollment by introducing an adaptive speaker embedding self-augmentation framework and a long-term embedding adaptation strategy. The method augments enrollment embeddings with a frame-level embedding extracted from mixed speech and refines these embeddings over multiple segments using a weighted update rule, all without retraining. Empirical results show notable improvements in recall, precision, and F1-score for short enrollment, with the long-term approach achieving parity with full enrollment after five iterations. The approach offers practical benefits for PVAD in real-world, wake-word-driven scenarios by enhancing personalization with limited enrollment data.

Abstract

Personal Voice Activity Detection (PVAD) is crucial for identifying target speaker segments in the mixture, yet its performance heavily depends on the quality of speaker embeddings. A key practical limitation is the short enrollment speech--such as a wake-up word--which provides limited cues. This paper proposes a novel adaptive speaker embedding self-augmentation strategy that enhances PVAD performance by augmenting the original enrollment embeddings through additive fusion of keyframe embeddings extracted from mixed speech. Furthermore, we introduce a long-term adaptation strategy to iteratively refine embeddings during detection, mitigating speaker temporal variability. Experiments show significant gains in recall, precision, and F1-score under short enrollment conditions, matching full-length enrollment performance after five iterative updates. The source code is available at https://anonymous.4open.science/r/ASE-PVAD-E5D6 .
Paper Structure (17 sections, 6 equations, 4 figures, 2 tables)

This paper contains 17 sections, 6 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Methods
  3. PVAD Backbone
  4. Speaker Embedding Self-Augmentation
  5. Self-Augmentation
  6. Long-term Speaker Embedding Adaptation
  7. Experimental Setup
  8. Datasets
  9. Implementation Details
  10. Results and Discussion
  11. Speaker Embedding Self-Augmentation
  12. Performance in Segment 1
  13. Performance in Segment 2
  14. Long-term Speaker Embedding Adaptation
  15. Performance in Segment 1-10
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Adaptive speaker embedding self-augmentation (N Iterations)
  • Figure 2: Architecture of adaptive speaker embedding self-augmentation for personal VAD. (a) illustrates our self-augmentation method, where $\mathit{E}_{\mathit{augmented}}$ ((1-N) denotes the iterations of self-augmentation, as illustrates in Fig. \ref{['fig1']}) is derived by similarity-guided fusion of the original enrollment embedding $\mathit{E}_{\mathit{enroll}}$ and the optimally matched frame-level embedding $\mathit{E}_{\mathit{selected}}$ from the mixed speech. (b) illustrates our PVAD 2.0 backbone architecture processing both the augmented speaker embedding $\mathit{E}_{\mathit{augmented}}$ and the acoustic features $\mathit{F}_{\mathit{mix}}$ extracted from the mixed speech.
  • Figure 3: Results for speaker embedding self-augmentation with 0.5s enrollment in segment 1-10 (where N=10, as defined in Fig. \ref{['fig1']}). Bars (left axis) show REC/PRE; lines (right axis) show F1; X-axis shows Segment 1-10. Experimental conditions: (a) Clean, Eq. \ref{['eq4']} fusion; (b) Noisy, Eq. \ref{['eq4']} fusion; (c) Clean, Eqs. \ref{['eq5']}--\ref{['eq6']} fusion; (d) Noisy, Eqs. \ref{['eq5']}--\ref{['eq6']} fusion. For comparison, the red dashed line in (c) and (d) replicates the F1 curve from (a) and (b), respectively.
  • Figure 4: (a) Ablation study of $\lambda$ (0.05, 0.1, 0.2, 0.3, 0.4) in Eq. \ref{['eq6']} for speaker embedding self-augmentation under 0.5s enrollment over 1-10 iterations (where N=10 in Fig. \ref{['fig1']}) in noisy condition. (b) PVAD performance visualization comparison: Original 0.5s enrollment vs. 5-iteration augmented embeddings (Eq. \ref{['eq4']} vs. Eqs. \ref{['eq5']}--\ref{['eq6']}).