Unimodal and Multimodal Sensor Fusion for Wearable Activity Recognition
Hymalai Bello
TL;DR
This paper addresses robust human activity recognition (HAR) by leveraging multimodal and multi-positional wearable sensing. It proposes HW/SW co-designs that fuse inertial, pressure-based (audio and atmospheric pressure), and textile capacitive modalities, with neural networks deployed on-edge for real-time inference. The key contributions include modular fusion architectures and demonstrations on devices such as wristbands, goggles, headwear, and clothing, achieving a low memory footprint ($\leq 2$ MB) to enable on-device deployment. The results show effective hand position tracking, facial/head pattern recognition, and body gesture recognition in realistic wearable scenarios, underscoring the practical impact for ubiquitous HAR. Future work will extend the fusion schemes to broader contexts and optimize for energy, latency, and robustness in real-world environments.
Abstract
Combining different sensing modalities with multiple positions helps form a unified perception and understanding of complex situations such as human behavior. Hence, human activity recognition (HAR) benefits from combining redundant and complementary information (Unimodal/Multimodal). Even so, it is not an easy task. It requires a multidisciplinary approach, including expertise in sensor technologies, signal processing, data fusion algorithms, and domain-specific knowledge. This Ph.D. work employs sensing modalities such as inertial, pressure (audio and atmospheric pressure), and textile capacitive sensing for HAR. The scenarios explored are gesture and hand position tracking, facial and head pattern recognition, and body posture and gesture recognition. The selected wearable devices and sensing modalities are fully integrated with machine learning-based algorithms, some of which are implemented in the embedded device, on the edge, and tested in real-time.