HyWA: Hypernetwork Weight Adapting Personalized Voice Activity Detection

TL;DR

This work tackles personalized voice activity detection (PVAD) by avoiding architectural changes to the VAD. It introduces HyWA, a hypernetwork-based weight adaptation method that generates a sparse delta $\Delta \mathbf w$ to a subset of the base VAD parameters $\mathbf w$, yielding $\mathcal{M}_{\mathbf w+\Delta \mathbf w}$ conditioned on a speaker embedding $\mathbf s$. Training jointly optimizes $\theta$, $\mathbf w$, and $\Delta \mathbf w$ using audio features $\mathbf a$ and ternary labels $y \in \{ns, tss, ntss\}$, evaluated with AP/mAP under clean and noisy conditions. Results on a LibriSpeech-based dataset show that HyWA outperforms FiLM and other conditioning methods, with robust improvements in seen and unseen noise, while maintaining deployment simplicity by reusing a single base VAD model.

Abstract

Personalized Voice Activity Detection (PVAD) systems activate only in response to a specific target speaker by incorporating speaker embeddings from enrollment utterances. Unlike existing methods that require architectural changes, such as FiLM layers, our approach employs a hypernetwork to modify the weights of a few selected layers within a standard voice activity detection (VAD) model. This enables speaker conditioning without changing the VAD architecture, allowing the same VAD model to adapt to different speakers by updating only a small subset of the layers. We propose HyWA-PVAD, a hypernetwork weight adaptation method, and evaluate it against multiple baseline conditioning techniques. Our comparison shows consistent improvements in PVAD performance. HyWA also offers practical advantages for deployment by preserving the core VAD architecture. Our new approach improves the current conditioning techniques in two ways: i) increases the mean average precision, ii) simplifies deployment by reusing the same VAD architecture.

Figures (3)

• Figure 1: An overview of the training pipeline for HyWA-PVAD. The hypernetwork $\mathcal{H}_\theta$ produces user-specific subset of VAD weights $\Delta \mathbf w$ based on speaker embedding $\mathbf s_i$. The VAD takes $\Delta \mathbf w$ along with audio features $\mathbf a$, and outputs PVAD labels $y$.
• Figure 2: An overview of the inference pipeline for HyWA-PVAD; where (a) shows the enrollment and deployment stages, executed once per user through cloud-device communication, and (b) illustrates the usage, executed on the device after enrollment and deployment stages.
• Figure 3: Speaker conditioning methods for PVAD. Each approach illustrates a distinct strategy for integrating speaker information with acoustic features to enable personalization.