MobiDiary: Autoregressive Action Captioning with Wearable Devices and Wireless Signals

TL;DR

MobiDiary addresses the challenge of describing daily activities from nonvisual, privacy-preserving sensor data by proposing a unified sensor encoder that treats IMU and WiFi signals as shared time-series reflections of human motion. The model uses patch-based local temporal tokens, placement embeddings, Conv-FFN blocks, and a Transformer-based autoregressive decoder to generate coherent natural language captions, trained with teacher forcing and a constrained vocabulary. Across four public datasets for IMU and WiFi modalities, MobiDiary achieves state-of-the-art results on captioning metrics and the semantic fidelity measure RMC, demonstrating strong generalization across modalities and sensor configurations. The work highlights the practical potential for interpretable, log-like activity descriptions suitable for health monitoring, smart homes, and downstream reasoning with LLMs, while acknowledging room for exploration in broader environments and real-time deployment.

Abstract

Human Activity Recognition (HAR) in smart homes is critical for health monitoring and assistive living. While vision-based systems are common, they face privacy concerns and environmental limitations (e.g., occlusion). In this work, we present MobiDiary, a framework that generates natural language descriptions of daily activities directly from heterogeneous physical signals (specifically IMU and Wi-Fi). Unlike conventional approaches that restrict outputs to pre-defined labels, MobiDiary produces expressive, human-readable summaries. To bridge the semantic gap between continuous, noisy physical signals and discrete linguistic descriptions, we propose a unified sensor encoder. Instead of relying on modality-specific engineering, we exploit the shared inductive biases of motion-induced signals--where both inertial and wireless data reflect underlying kinematic dynamics. Specifically, our encoder utilizes a patch-based mechanism to capture local temporal correlations and integrates heterogeneous placement embedding to unify spatial contexts across different sensors. These unified signal tokens are then fed into a Transformer-based decoder, which employs an autoregressive mechanism to generate coherent action descriptions word-by-word. We comprehensively evaluate our approach on multiple public benchmarks (XRF V2, UWash, and WiFiTAD). Experimental results demonstrate that MobiDiary effectively generalizes across modalities, achieving state-of-the-art performance on captioning metrics (e.g., BLEU@4, CIDEr, RMC) and outperforming specialized baselines in continuous action understanding.

Figures (6)

• Figure 1: A user moves around a home environment. MobiDiary captures the sensor signal sequence (originating from either worn IMU-equipped devices or ambient WiFi signals) and generates natural language descriptions of the user's activities. These textual summaries can serve as interpretable logs and support downstream tasks such as activity retrieval, monitoring, and assistive reasoning, especially when integrated with LLMs to enhance contextual understanding and reasoning capabilities.
• Figure 2: Overview of MobiDiary. During training, the input IMU or Wi-Fi sequence is encoded by the Sensor Encoder to extract motion features, while the ground-truth text description is processed by a text encoder. The fused features are then fed into a Language Generation Network to produce the action caption. During inference, the Text Encoder is initialized with a start-of-sequence token (<sos>), and action captions are generated in an Autoregressive manner through next-word prediction.
• Figure 3: Sinusoidal Position Encoding. The decline of two positions' similarity of $\langle p_m, p_n \rangle$ as $|m-n|$ increases for different values of $\theta_t$.
• Figure 4: During training, we employ the Teacher Forcing strategy, whereas during inference, we adopt an Autoregressive generation approach.
• Figure 5: Qualitative results of the MobiDiary model on IMU (a, b) and WiFi (c, d) datasets. The model generated accurate descriptions on the XRF V2 (IMU) (a), UWash (b), and WiFiTAD (d) datasets, but struggled with abstract signals in the XRF V2 (WiFi) (c) dataset, confusing "cuts fruit" with "writes".