Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encoder

TL;DR

The paper tackles efficient streaming keyword spotting by introducing input-dependent dynamic depth in a conformer encoder equipped with trainable binary gates that decide, at module granularity, whether to skip computations. The gating mechanism computes $p_{keep}$ via $p_g(x)=\mathrm{Softmax}(W_g\cdot\bar{x}+b_g)$ and uses the Gumbel-Softmax trick during training with a gate-regularization term to encourage sparsity, while inference relies on a simple threshold to binarize decisions; a pretraining strategy is also employed. The authors integrate this gated conformer into a vision-inspired KWS pipeline with detection, classification, and CenterNet-inspired localization heads, using max-pooling to handle variable keyword lengths. Experiments on Librispeech Top-1000 keywords and Google Speech Commands with noise show substantial compute reductions (up to ~97% MACs skipped on non-speech) with minimal or no loss in accuracy, and fewer parameters than the BC-ResNet baseline, highlighting practical benefits for always-on keyword spotting. Overall, the approach delivers a more energy-efficient, streaming KWS system suitable for portable devices by adaptively reducing processing in response to input content while preserving detection and localization performance.

Abstract

Using a vision-inspired keyword spotting framework, we propose an architecture with input-dependent dynamic depth capable of processing streaming audio. Specifically, we extend a conformer encoder with trainable binary gates that allow us to dynamically skip network modules according to the input audio. Our approach improves detection and localization accuracy on continuous speech using Librispeech top-1000 most frequent words while maintaining a small memory footprint. The inclusion of gates also reduces the average amount of processing without affecting the overall performance. These benefits are shown to be even more pronounced using the Google speech commands dataset placed over background noise where up to 97% of the processing is skipped on non-speech inputs, therefore making our method particularly interesting for an always-on keyword spotter.

Figures (1)

• Figure 1: Keyword spotting pipeline in training mode.