Speech Understanding on Tiny Devices with A Learning Cache

TL;DR

This work tackles spoken language understanding (SLU) on ultra-low-resource devices by introducing SpeechCache (SC), a two-level on-device cache that reuses recent inferences and offloads only novel inputs to the cloud. By matching inputs at two representations—sound-unit sequences (L1) and phoneme sequences (L2)—SC achieves robust, low-cost on-device inference and leverages cloud feedback to continuously personalize local feature extractors. The approach yields substantial latency reductions (up to ~$80\%$) with a sub-2 MB memory footprint on an MCU, resolving 45–90% of inputs on-device on realistic benchmarks while maintaining competitive accuracy. Through a suite of optimizations, online learning, and model ensembling, SC demonstrates practical edge-cloud SLU with resilience to noise, cold starts, and multi-user scenarios, offering significant reductions in cloud usage and latency for smart devices. The work advances edge AI by showing how temporal locality in user speech can be harnessed with hierarchical representations and online adaptation to deliver near-SOTA SLU on tiny hardware.

Abstract

This paper addresses spoken language understanding (SLU) on microcontroller-like embedded devices, integrating on-device execution with cloud offloading in a novel fashion. We leverage temporal locality in the speech inputs to a device and reuse recent SLU inferences accordingly. Our idea is simple: let the device match incoming inputs against cached results, and only offload inputs not matched to any cached ones to the cloud for full inference. Realization of this idea, however, is non-trivial: the device needs to compare acoustic features in a robust yet low-cost way. To this end, we present SpeechCache (or SC), a speech cache for tiny devices. It matches speech inputs at two levels of representations: first by sequences of clustered raw sound units, then as sequences of phonemes. Working in tandem, the two representations offer complementary tradeoffs between cost and efficiency. To boost accuracy even further, our cache learns to personalize: with the mismatched and then offloaded inputs, it continuously finetunes the device's feature extractors with the assistance of the cloud. We implement SC on an off-the-shelf STM32 microcontroller. The complete implementation has a small memory footprint of 2MB. Evaluated on challenging speech benchmarks, our system resolves 45%-90% of inputs on device, reducing the average latency by up to 80% compared to offloading to popular cloud speech recognition services. The benefit brought by our proposed SC is notable even in adversarial settings - noisy environments, cold cache, or one device shared by a number of users.

Figures (8)

• Figure 1: Our system in the design space of speech understanding systems.
• Figure 2: Typical models and parameter sizes for SLU. (Top) a large, conventional model huang2023leveraging; (Bottom) an end-to-end model compressed for MCU, which often can only understand simple commands
• Figure 3: Raw speech waveforms often exhibit strong variations. Three waveforms of the same sentence ("wake me up at 5am this week"), uttered by the same speaker, from SLURP are shown. First two are recorded in far field conditions while the third is recorded in close range (headset).
• Figure 4: The design of sound unit cache (L1)
• Figure 5: The design of phoneme cache (L2).