iRoCo: Intuitive Robot Control From Anywhere Using a Smartwatch

TL;DR

iRoCo addresses the challenge of enabling robot control anywhere using consumer wearables by fusing smartwatch and smartphone data through a Differentiable Ensemble Kalman Filter to produce robust pose estimates and a tailored control modality for end-effector control. The approach enables intuitive teleoperation and outdoor drone piloting from anywhere, achieving real-time performance and competitive or better task efficiency while reducing subjective workload. The work demonstrates strong generalization to unconstrained environments and provides an end-to-end trainable pipeline with detailed data collection and evaluation across two robot domains. The authors release code at www.github.com/wearable-motion-capture to facilitate adoption and extension in HRI contexts.

Abstract

This paper introduces iRoCo (intuitive Robot Control) - a framework for ubiquitous human-robot collaboration using a single smartwatch and smartphone. By integrating probabilistic differentiable filters, iRoCo optimizes a combination of precise robot control and unrestricted user movement from ubiquitous devices. We demonstrate and evaluate the effectiveness of iRoCo in practical teleoperation and drone piloting applications. Comparative analysis shows no significant difference between task performance with iRoCo and gold-standard control systems in teleoperation tasks. Additionally, iRoCo users complete drone piloting tasks 32\% faster than with a traditional remote control and report less frustration in a subjective load index questionnaire. Our findings strongly suggest that iRoCo is a promising new approach for intuitive robot control through smartwatches and smartphones from anywhere, at any time. The code is available at www.github.com/wearable-motion-capture

Figures (7)

• Figure 1: A) We present an intuitive Robot Control (iRoCo) framework, which achieves robust body orientation and arm pose estimations from a smartwatch and a smartphone. B) The system allows teleoperation through video streaming to a smartphone, enabling tasks such as pick-and-place operations. C) We demonstrate iRoCo's potential for intuitive drone piloting.
• Figure 2: Left: Data collection to record ground truth OptiTrack poses together with smartwatch and smartphone sensor data. Right: Starting our app on the watch calibrates the watch and phone orientation. The start pose sets the local coordinate system and defines the forward direction.
• Figure 3: The DEnKF model structure. In the Prediction Step, the stochastic Transition Model forwards the ensemble ${\bf X}_{{t-N}:{t-1}}$ one step in time (${\bf \tilde{X}}_t$). In the following Update Step, the Sensor Model projects raw observations to the observation space and the Observation Model projects ${\bf \tilde{X}}_t$ to the observation space, such that the KF Update corrects ${\bf \tilde{X}}_t$ and we obtain ${\bf X}_t$.
• Figure 4: We propose a control modality where the body forward-facing direction together with the estimated wrist position on the sagittal plane determine the final end-effector position relative to the robot's base. This allows fine-grained control as demonstrated by writing the letters A, M, B, Z with the robot end-effector.
• Figure 5: Top: Prediction error distributions of DEnKF on the test dataset. Bottom: An example for widening ensemble distributions, i.e., higher uncertainty, when the user moves fast. Ensemble members are colored according to their distance from the mean. GT is the Ground Truth.