AnalyticKWS: Towards Exemplar-Free Analytic Class Incremental Learning for Small-footprint Keyword Spotting

TL;DR

This work tackles exemplar-free continual learning for small-footprint keyword spotting on edge devices, addressing catastrophic forgetting without storing past data. It presents AnalyticKWS, which freezes a pretrained CNN feature extractor, uses a feature expansion to form $\mathbf{S}_0'$ and solves the classifier weights analytically via $\widehat{\theta}_{cls}^{(0)} = (\mathbf{S}_0'^{\top} \mathbf{S}_0' + \gamma I)^{-1} \mathbf{S}_0'^{\top} \mathbf{y}_0$, and then updates weights and an AFAM matrix recursively with Woodbury-derived formulas to incorporate new keywords in one epoch per task. The methodology achieves near-joint accuracy with minimal forgetting across multiple KWS benchmarks, while reducing training time and memory by avoiding gradient updates and exemplar storage, making it suitable for privacy-preserving, resource-constrained on-device deployment. Empirical results on GSC v1/v2 and SC-100 demonstrate strong ACC, low BWT, and substantial efficiency gains over exemplar-based baselines, highlighting AnalyticKWS’s practical impact for edge keyword spotting.

Abstract

Keyword spotting (KWS) offers a vital mechanism to identify spoken commands in voice-enabled systems, where user demands often shift, requiring models to learn new keywords continually over time. However, a major problem is catastrophic forgetting, where models lose their ability to recognize earlier keywords. Although several continual learning methods have proven their usefulness for reducing forgetting, most existing approaches depend on storing and revisiting old data to combat catastrophic forgetting. Though effective, these methods face two practical challenges: 1) privacy risks from keeping user data and 2) large memory and time consumption that limit deployment on small devices. To address these issues, we propose an exemplar-free Analytic Continual Learning (AnalyticKWS) method that updates model parameters without revisiting earlier data. Inspired by efficient learning principles, AnalyticKWS computes a closed-form analytical solution for model updates and requires only a single epoch of adaptation for incoming keywords. AnalyticKWS demands fewer computational resources by avoiding gradient-based updates and does not store old data. By eliminating the need for back-propagation during incremental learning, the model remains lightweight and efficient. As a result, AnalyticKWS meets the challenges mentioned earlier and suits resource-limited settings well. Extensive experiments on various datasets and settings show that AnalyticKWS consistently outperforms existing continual learning methods.

Figures (4)

• Figure 1: An overview of the AnalyticKWS method: (a) Train the whole model on the first task for multiple epochs to get a strong feature extractor, then (b) Apply analytic re-alignment for one epoch to increase the pre-classifier feature dimension. Next, proceed to the incremental keywords stage, where the model trains for one epoch per new task, assisted by a correlation matrix AFAM (Eq. \ref{['eq6']}) that encodes past knowledge. This process enables the model to learn new tasks while preserving previously acquired information.
• Figure 2: Task-wise performance comparison of different methods with 500 buffer size.
• Figure 3: Task-wise accuracy on GSC-v2 with 11 tasks.
• Figure 4: Task-wise accuracy on GSC-v2 with 21 tasks.

Theorems & Definitions (1)

• proof