EgoLog: Ego-Centric Fine-Grained Daily Log with Ubiquitous Wearables
Lixing He, Bufang Yang, Di Duan, Zhenyu Yan, Guoliang Xing
TL;DR
EgoLog tackles the challenge of fine-grained daily logging with egocentric wearables by integrating audio and IMU data through temporal and spatial understanding, enhanced by scenario-aware HAR and LLM-assisted refinement. The approach demonstrates robust multimodal fusion, effective sound localization with movement compensation, and edge-cloud collaboration to improve scenario recognition while maintaining on-device efficiency. Evaluations on public (Ego4D, EgoADL, SAMOSA) and self-collected datasets show significant gains in activity (≈12%) and scenario recognition (≈15%), with favorable user-perceived usefulness and acceptable overhead. This work advances practical, privacy-conscious, long-term daily logging with ubiquitous wearables for healthcare and continuous monitoring.
Abstract
Despite advances in human activity recognition (HAR) with different modalities, a precise, robust, and accurate daily log system is not yet available. Current solutions primarily rely on controlled, lab-based data collection, which limits their real-world applicability. The challenges towards a fine-grained daily log are 1) contextual awareness, 2) spatial awareness, and 3) effective fusion of multi-modal sensor data. To solve them, we propose EgoLog, which integrates effective audio-IMU fusion for daily log with ubiquitous wearables. Our approach first fuses audio and IMU data from two perspectives: temporal understanding and spatial understanding. We extract scenario-level features and aggregate them in the time dimension, while using motion compensation to enhance the performance of sound source localization. The knowledge obtained from these steps is then integrated into a multi-modal HAR framework. Here, the scenario provides prior knowledge, and the spatial location helps differentiate the user from the background. Furthermore, we integrate a LLM to enhance scenario recognition through logical reasoning. The knowledge derived from the LLM is subsequently transferred back to the local device to enable efficient, on-device inference. Evaluated on both public and self-collected dataset, EgoLog achieves effective multimodal fusion for both activity and scenraio recognition, outperforms the baseline by 12% and 15%, respectively.