ERSAM: Neural Architecture Search For Energy-Efficient and Real-Time Social Ambiance Measurement

TL;DR

ERSAM addresses the need for on-device, privacy-preserving social ambiance measurement by balancing accuracy, latency, and energy on mobile hardware. It introduces two key innovations: a SAM-specific hardware-aware NAS search space and an uncertainty-driven efficient knowledge distillation scheme, integrated into a weight-sharing NAS framework. Empirical results on LibriSpeech-SAM show ERSAM achieves $14.3\%$ error with latency $\approx 0.05\ \text{s}$ for a 5-second segment and energy $40\ \text{mW}\cdot 12\ \text{h}$ on a Pixel 3, outperforming previous baselines in the accuracy vs hardware cost frontier. The approach generalizes to real-world noisy and device-diverse scenarios, highlighting practical potential for ubiquitous on-device SAM in mental health and human-centered IoT applications.

Abstract

Social ambiance describes the context in which social interactions happen, and can be measured using speech audio by counting the number of concurrent speakers. This measurement has enabled various mental health tracking and human-centric IoT applications. While on-device Socal Ambiance Measure (SAM) is highly desirable to ensure user privacy and thus facilitate wide adoption of the aforementioned applications, the required computational complexity of state-of-the-art deep neural networks (DNNs) powered SAM solutions stands at odds with the often constrained resources on mobile devices. Furthermore, only limited labeled data is available or practical when it comes to SAM under clinical settings due to various privacy constraints and the required human effort, further challenging the achievable accuracy of on-device SAM solutions. To this end, we propose a dedicated neural architecture search framework for Energy-efficient and Real-time SAM (ERSAM). Specifically, our ERSAM framework can automatically search for DNNs that push forward the achievable accuracy vs. hardware efficiency frontier of mobile SAM solutions. For example, ERSAM-delivered DNNs only consume 40 mW x 12 h energy and 0.05 seconds processing latency for a 5 seconds audio segment on a Pixel 3 phone, while only achieving an error rate of 14.3% on a social ambiance dataset generated by LibriSpeech. We can expect that our ERSAM framework can pave the way for ubiquitous on-device SAM solutions which are in growing demand.

Figures (4)

• Figure 1: DNNs searched by our ERSAM outperform state-of-the-art DNN-based SAM: achieving the best accuracy vs. hardware efficiency trade-offs while meeting the requirements of being energy-efficient and achieving real-time latency. Here a smaller volume of each data point's 3D cuboid corresponds to a better accuracy vs. hardware efficiency trade-off.
• Figure 2: Overview of our proposed ERSAM framework.
• Figure 3: Latency of different operators in wav2vec schneider2019wav2vec profiled on a Pixel 3 pixel3 phone.
• Figure 4: The percentage of cases when varying the smaller model's uncertainties (AC: both correct, LC: only the larger model correct, SC: only the smaller model correct, and AW: both wrong).