"It is okay to be uncommon": Quantizing Sound Event Detection Networks on Hardware Accelerators with Uncommon Sub-Byte Support
Yushu Wu, Xiao Quan, Mohammad Rasool Izadi, Chuan-Che Huang
TL;DR
This work addresses the challenge of running sound event detection on memory- and energy-constrained headphones by leveraging accelerators that support both common and uncommon sub-byte quantization and a differentiable NAS method (FracBit) to per-layer bitwidths under a storage constraint. The authors validate on two SED tasks—generic classification and few-shot embedding-based detection—using DCRNN and HiSSNet, and demonstrate substantial on-device gains: roughly 54–69% memory reduction, 45–47% latency reduction, and 53–69% energy reduction compared with 8-bit baselines, while preserving floating-point accuracy. A key finding is that FracBit consistently outperforms fixed-bitwidth quantization in Pareto-optimal memory-accuracy trade-offs and that the searched models naturally adopt uncommon bitwidths per layer, aligned with hardware capabilities like NE16. The results provide practical guidance for designing edge audio ML pipelines and hardware accelerators for on-device SED with mixed-precision quantization.
Abstract
If our noise-canceling headphones can understand our audio environments, they can then inform us of important sound events, tune equalization based on the types of content we listen to, and dynamically adjust noise cancellation parameters based on audio scenes to further reduce distraction. However, running multiple audio understanding models on headphones with a limited energy budget and on-chip memory remains a challenging task. In this work, we identify a new class of neural network accelerators (e.g., NE16 on GAP9) that allows network weights to be quantized to different common (e.g., 8 bits) and uncommon bit-widths (e.g., 3 bits). We then applied a differentiable neural architecture search to search over the optimal bit-widths of a network on two different sound event detection tasks with potentially different requirements on quantization and prediction granularity (i.e., classification vs. embeddings for few-shot learning). We further evaluated our quantized models on actual hardware, showing that we reduce memory usage, inference latency, and energy consumption by an average of 62%, 46%, and 61% respectively compared to 8-bit models while maintaining floating point performance. Our work sheds light on the benefits of such accelerators on sound event detection tasks when combined with an appropriate search method.