IMUFace: Towards Always-On 3D Facial Reconstruction via Earphone Inertial Sensing

TL;DR

IMUFace tackles privacy and practicality limitations of camera-based 3D facial reconstruction by leveraging inertial sensing in wireless earphones. It introduces IMUTwinTrans, a ConvTransformer-based network that fuses temporal and frequency features from earphone IMUs to predict 51 facial landmarks, which are then fitted to the FLAME head model for continuous 3D reconstruction. The system operates at 30 Hz with only 58 mW of power, and a 12-participant study shows an average landmark error of 2.21 mm (NME 3.40%), with improvements via domain adaptation. This approach offers a privacy-preserving, low-power alternative for real-time facial expression capture with broad implications for HCI, VR/AR, healthcare, and automotive safety.

Abstract

The potential of facial expression reconstruction technology is significant, with applications in various fields such as human-computer interaction, affective computing, and virtual reality. Recent studies have proposed using ear-worn devices for facial expression reconstruction to address the environmental limitations and privacy concerns associated with traditional camera-based methods. However, these approaches still require improvements in terms of aesthetics and power consumption. This paper introduces a system called IMUFace. It uses inertial measurement units (IMUs) embedded in wireless earphones to detect subtle ear movements caused by facial muscle activities, allowing for covert and low-power facial reconstruction. A user study involving 12 participants was conducted, and a deep learning model named IMUTwinTrans was proposed. The results show that IMUFace can accurately predict users' facial landmarks with a precision of 2.21 mm, using only five minutes of training data. The predicted landmarks can be utilized to reconstruct a three-dimensional facial model. IMUFace operates at a sampling rate of 30 Hz with a relatively low power consumption of 58 mW. The findings presented in this study demonstrate the real-world applicability of IMUFace and highlight potential directions for further research to facilitate its practical adoption.

Figures (9)

• Figure 1: Demonstration of continuous 3D facial reconstruction of IMUFace.
• Figure 2: Overview of the IMUFace system.
• Figure 3: Device Prototype: Composed of a pair of earbuds and XIAO nRF52840 Sense MCU.
• Figure 4: Illustration of facial landmark alignment. (a) Before landmark alignment. (b) After landmark alignment.
• Figure 5: Illustration of the model structure.