Human Motion Estimation with Everyday Wearables

TL;DR

EveryWear tackles the challenge of practical full-body motion capture in daily life using everyday wearables by coupling egocentric RGB streams from three glasses cameras with consumer IMUs through a teacher–student distillation framework. A monocular SLAM module supplies head pose to mitigate drift, while training on real-world data from the Ego-Elec dataset eliminates the sim-to-real gap typical of synthetic-data approaches. The approach achieves state-of-the-art accuracy, reporting an MPJPE of $8.459$ cm and an MPJVE of $10.627$ cm on Ego-Elec, and demonstrates robust performance under occlusions via cross-modal compensation. This work advances XR interaction and telepresence by enabling calibration-free, practical motion capture with widely available consumer devices, supported by a large real-world dataset for future research.

Abstract

While on-body device-based human motion estimation is crucial for applications such as XR interaction, existing methods often suffer from poor wearability, expensive hardware, and cumbersome calibration, which hinder their adoption in daily life. To address these challenges, we present EveryWear, a lightweight and practical human motion capture approach based entirely on everyday wearables: a smartphone, smartwatch, earbuds, and smart glasses equipped with one forward-facing and two downward-facing cameras, requiring no explicit calibration before use. We introduce Ego-Elec, a 9-hour real-world dataset covering 56 daily activities across 17 diverse indoor and outdoor environments, with ground-truth 3D annotations provided by the motion capture (MoCap), to facilitate robust research and benchmarking in this direction. Our approach employs a multimodal teacher-student framework that integrates visual cues from egocentric cameras with inertial signals from consumer devices. By training directly on real-world data rather than synthetic data, our model effectively eliminates the sim-to-real gap that constrains prior work. Experiments demonstrate that our method outperforms baseline models, validating its effectiveness for practical full-body motion estimation.

Figures (8)

• Figure 1: EveryWear is a novel human motion capture approach based on a series of lightweight everyday wearables: a smartphone, smartwatch, earbuds, and smart glasses equipped with one forward-facing and two downward-facing cameras.
• Figure 2: System illustration. The system comprises smart glasses (with onboard Raspberry Pi), smartphone, smartwatch, and earbuds. An XSens mocap system provides annotations.
• Figure 3: Pipeline. Our method takes egocentric images (three cameras) and IMU signals (everyday wearables) as input. We employ SLAM (\ref{['sec:slam']}) for head pose estimation, then use a teacher-student framework (\ref{['sec:ts_model']}) with shared visual feature encoder and separate IMU feature encoders, followed by bidirectional LSTM for temporal fusion and motion prediction.
• Figure 4: Qualitative comparison. This figure presents a comparison against baseline methods on Ego-Elec, where IMUPoser serves as the IMU-only baseline and Fish2Mesh represents the camera-only baseline. We visualize per-vertex SMPL error using a color map ranging from $0$ to $0.3$ m (white: low error, red: high error). Our method achieves consistently lower errors across diverse activities and maintains robustness under challenging occlusion scenarios.
• Figure 5: Failure cases. We show rare failure scenarios where loose attachment of consumer devices (phone shifting in pocket, watch with play) introduces sensor noise and instability.