Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ubiquitous Robot Control Through Multimodal Motion Capture Using Smartwatch and Smartphone Data

Fabian C Weigend, Neelesh Kumar, Oya Aran, Heni Ben Amor

TL;DR

The paper addresses the challenge of enabling robot control in ubiquitous settings using consumer wearables by introducing WearMoCap, an open-source library with three multimodal pose-estimation modes. The approach combines $\mathbf{q}_{la}$ and $\mathbf{q}_{ua}$ estimates from LSTM-based pipelines (Watch Only and Upper Arm) and a Differentiable Ensemble Kalman Filter leveraging $\mathbf{q}_{hi}$ for the Pocket mode, with forward kinematics to produce end-effector positions. Real-robot validation on UR5 across Handover and Intervention tasks shows sub-$2\,\mathrm{cm}$ accuracy for the Upper Arm mode and clear trade-offs among modes, highlighting the balance between precision and ubiquity. The work demonstrates practical, open-source multimodal motion capture for robot teleoperation in everyday environments and informs mode selection for specific tasks, with the repository available at $www.github.com/wearable-motion-capture$.

Abstract

We present an open-source library for seamless robot control through motion capture using smartphones and smartwatches. Our library features three modes: Watch Only Mode, enabling control with a single smartwatch; Upper Arm Mode, offering heightened accuracy by incorporating the smartphone attached to the upper arm; and Pocket Mode, determining body orientation via the smartphone placed in any pocket. These modes are applied in two real-robot tasks, showcasing placement accuracy within 2 cm compared to a gold-standard motion capture system. WearMoCap stands as a suitable alternative to conventional motion capture systems, particularly in environments where ubiquity is essential. The library is available at: www.github.com/wearable-motion-capture.

Ubiquitous Robot Control Through Multimodal Motion Capture Using Smartwatch and Smartphone Data

TL;DR

The paper addresses the challenge of enabling robot control in ubiquitous settings using consumer wearables by introducing WearMoCap, an open-source library with three multimodal pose-estimation modes. The approach combines and estimates from LSTM-based pipelines (Watch Only and Upper Arm) and a Differentiable Ensemble Kalman Filter leveraging for the Pocket mode, with forward kinematics to produce end-effector positions. Real-robot validation on UR5 across Handover and Intervention tasks shows sub- accuracy for the Upper Arm mode and clear trade-offs among modes, highlighting the balance between precision and ubiquity. The work demonstrates practical, open-source multimodal motion capture for robot teleoperation in everyday environments and informs mode selection for specific tasks, with the repository available at .

Abstract

We present an open-source library for seamless robot control through motion capture using smartphones and smartwatches. Our library features three modes: Watch Only Mode, enabling control with a single smartwatch; Upper Arm Mode, offering heightened accuracy by incorporating the smartphone attached to the upper arm; and Pocket Mode, determining body orientation via the smartphone placed in any pocket. These modes are applied in two real-robot tasks, showcasing placement accuracy within 2 cm compared to a gold-standard motion capture system. WearMoCap stands as a suitable alternative to conventional motion capture systems, particularly in environments where ubiquity is essential. The library is available at: www.github.com/wearable-motion-capture.
Paper Structure (12 sections, 3 figures, 1 table)

This paper contains 12 sections, 3 figures, 1 table.

Table of Contents

  1. INTRODUCTION
  2. WearMoCap MODES
  3. Watch Only
  4. Upper Arm
  5. Pocket
  6. REAL-ROBOT TASKS
  7. Handover
  8. Intervention
  9. Results And Conclusions
  10. Watch Only
  11. Upper Arm
  12. Pocket

Figures (3)

  • Figure 1: Our three WearMoCap modes track the arm pose with varying accuracy and flexibility. We evaluated all modes in real-robot tasks.
  • Figure 2: Handover Task: Human subjects used WearMoCap to perform handovers with a UR5. A voice command completed the handover and the robot let go of the cube.
  • Figure 3: Intervention Task: The UR5 sorted the green and red cubes into the bin. Human subjects use WearMoCap to interrupt the robot and to place the green cube at a target location.