Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-supervised Pretraining for Robust Personalized Voice Activity Detection in Adverse Conditions

Holger Severin Bovbjerg, Jesper Jensen, Jan Østergaard, Zheng-Hua Tan

TL;DR

This work tackles robust personalized voice activity detection in noisy environments by leveraging self-supervised pretraining on large unlabeled data using Autoregressive Predictive Coding (APC) to pretrain a compact LSTM encoder. A denoising variant (DN-APC) is proposed to further enhance robustness by learning to recover clean future frames from noisy inputs, and the pretrained encoder is fine-tuned for the target task with a fixed speaker-embedding model. Across clean and various noisy test conditions, APC-based pretraining improves mean average precision ($mAP$) over a purely supervised baseline, with DN-APC+MTR delivering the strongest performance, including substantial gains when training data is limited (e.g., LibriLight 10h). The results suggest that self-supervised pretraining can meaningfully boost robustness and performance for practical personalized VAD systems, enabling better operation in real-world adverse environments without requiring large labeled datasets.

Abstract

In this paper, we propose the use of self-supervised pretraining on a large unlabelled data set to improve the performance of a personalized voice activity detection (VAD) model in adverse conditions. We pretrain a long short-term memory (LSTM)-encoder using the autoregressive predictive coding (APC) framework and fine-tune it for personalized VAD. We also propose a denoising variant of APC, with the goal of improving the robustness of personalized VAD. The trained models are systematically evaluated on both clean speech and speech contaminated by various types of noise at different SNR-levels and compared to a purely supervised model. Our experiments show that self-supervised pretraining not only improves performance in clean conditions, but also yields models which are more robust to adverse conditions compared to purely supervised learning.

Self-supervised Pretraining for Robust Personalized Voice Activity Detection in Adverse Conditions

TL;DR

This work tackles robust personalized voice activity detection in noisy environments by leveraging self-supervised pretraining on large unlabeled data using Autoregressive Predictive Coding (APC) to pretrain a compact LSTM encoder. A denoising variant (DN-APC) is proposed to further enhance robustness by learning to recover clean future frames from noisy inputs, and the pretrained encoder is fine-tuned for the target task with a fixed speaker-embedding model. Across clean and various noisy test conditions, APC-based pretraining improves mean average precision () over a purely supervised baseline, with DN-APC+MTR delivering the strongest performance, including substantial gains when training data is limited (e.g., LibriLight 10h). The results suggest that self-supervised pretraining can meaningfully boost robustness and performance for practical personalized VAD systems, enabling better operation in real-world adverse environments without requiring large labeled datasets.

Abstract

In this paper, we propose the use of self-supervised pretraining on a large unlabelled data set to improve the performance of a personalized voice activity detection (VAD) model in adverse conditions. We pretrain a long short-term memory (LSTM)-encoder using the autoregressive predictive coding (APC) framework and fine-tune it for personalized VAD. We also propose a denoising variant of APC, with the goal of improving the robustness of personalized VAD. The trained models are systematically evaluated on both clean speech and speech contaminated by various types of noise at different SNR-levels and compared to a purely supervised model. Our experiments show that self-supervised pretraining not only improves performance in clean conditions, but also yields models which are more robust to adverse conditions compared to purely supervised learning.
Paper Structure (13 sections, 2 equations, 2 figures, 5 tables)

This paper contains 13 sections, 2 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology and Data Sets
  3. Personalized VAD model
  4. Autoregressive predictive coding
  5. Data sets
  6. Supervised baseline
  7. Pretraining and fine-tuning
  8. Multistyle training
  9. Metrics
  10. Results
  11. Clean conditions
  12. Adverse conditions
  13. Conclusions

Figures (2)

  • Figure 1: Overview of the personalized VAD system model used in this work. We only train the blue boxes, while we use an already trained d-vector model with fixed weights.
  • Figure 2: Overview of the APC framework and DenoisingAPC variant. Note that noise is only added for DenoisingAPC, while this is skipped for standard APC.