VibOmni: Towards Scalable Bone-conduction Speech Enhancement on Earables
Lixing He, Yunqi Guo, Haozheng Hou, Zhenyu Yan
TL;DR
VibOmni tackles robust speech enhancement for earables by fusing audio with bone-conduction vibration using IMUs, enabling real-time processing in noisy environments. It introduces a two-branch encoder-decoder network, Bone Conduction Function–based data augmentation, and a multi-modal SNR estimator to support continual learning and adaptive inference without on-device back-propagation. The approach yields substantial gains in PESQ, SNR, and WER across real-user datasets, while maintaining low latency suitable for mobile devices, and is validated by a user study showing strong preference over baselines. The work demonstrates practical pathways for scalable, privacy-conscious speech enhancement on diverse head-worn platforms through data augmentation, adaptive processing, and cross-modal fusion.
Abstract
Earables, such as True Wireless Stereo earphones and VR/AR headsets, are increasingly popular, yet their compact design poses challenges for robust voice-related applications like telecommunication and voice assistant interactions in noisy environments. Existing speech enhancement systems, reliant solely on omnidirectional microphones, struggle with ambient noise like competing speakers. To address these issues, we propose VibOmni, a lightweight, end-to-end multi-modal speech enhancement system for earables that leverages bone-conducted vibrations captured by widely available Inertial Measurement Units (IMUs). VibOmni integrates a two-branch encoder-decoder deep neural network to fuse audio and vibration features. To overcome the scarcity of paired audio-vibration datasets, we introduce a novel data augmentation technique that models Bone Conduction Functions (BCFs) from limited recordings, enabling synthetic vibration data generation with only 4.5% spectrogram similarity error. Additionally, a multi-modal SNR estimator facilitates continual learning and adaptive inference, optimizing performance in dynamic, noisy settings without on-device back-propagation. Evaluated on real-world datasets from 32 volunteers with different devices, VibOmni achieves up to 21% improvement in Perceptual Evaluation of Speech Quality (PESQ), 26% in Signal-to-Noise Ratio (SNR), and about 40% WER reduction with much less latency on mobile devices. A user study with 35 participants showed 87% preferred VibOmni over baselines, demonstrating its effectiveness for deployment in diverse acoustic environments.