Enhancing Few-shot Keyword Spotting Performance through Pre-Trained Self-supervised Speech Models

TL;DR

This work tackles the challenge of few-shot keyword spotting on battery-powered edge devices by integrating self-supervised speech models into a teacher–student framework. A Wav2Vec 2.0–based teacher employs dimensionality reduction and the Sub-center ArcFace loss to enhance inter-class separability, while a lightweight ResNet15 student is trained via knowledge distillation to operate efficiently on edge hardware. Across MSWC and Google Speech Commands, the approach yields significant gains, notably raising 10-shot accuracy on GSC from $33.4\%$ to $74.1\%$ at a 1% false alarm rate, and demonstrates robust cross-domain performance when using the KD+$\mathcal{L}_{\text{subcenter}}$ strategy. The results highlight the practical potential of SSL-based representations for on-device FS-KWS, with implications for scalable, low-power wake-word systems and real-world keyword customization.

Abstract

Keyword Spotting plays a critical role in enabling hands-free interaction for battery-powered edge devices. Few-Shot Keyword Spotting (FS-KWS) addresses the scalability and adaptability challenges of traditional systems by enabling recognition of custom keywords with only a few examples. However, existing FS-KWS systems achieve subpar accuracy at desirable false acceptance rates, particularly in resource-constrained edge environments. To address these issues, we propose a training scheme that leverages self-supervised learning models for robust feature extraction, dimensionality reduction, and knowledge distillation. The teacher model, based on Wav2Vec 2.0 is trained using Sub-center ArcFace loss, which enhances inter-class separability and intra-class compactness. To enable efficient deployment on edge devices, we introduce attention-based dimensionality reduction and train a standard lightweight ResNet15 student model. We evaluate the proposed approach on the English portion of the Multilingual Spoken Words Corpus (MSWC) and the Google Speech Commands (GSC) datasets. Notably, the proposed training method improves the 10-shot classification accuracy from 33.4% to 74.1% on 11 classes at 1% false alarm accuracy on the GSC dataset, thus making it significantly better-suited for a real use case scenario.

Figures (3)

• Figure 1: Proposed approach for edge FS-KWS model training
• Figure 2: Proposed Teacher Representation Model with (a) simple pooling encoder, (b) attention encoder dimensionality reduction models
• Figure 3: Performance of (a) Teacher models on the MSWC dataset under 1-shot setting for different dimensionality reduction architectures (b) Student models on the MSWC dataset under 1-shot setting (c) Student models on the GSC dataset under 10-shot setting