Wrist2Finger: Sensing Fingertip Force for Force-Aware Hand Interaction with a Ring-Watch Wearable

TL;DR

Wrist2Finger introduces a minimalist ring-watch wearable that enables force-aware hand interaction by fusing a single IMU ring with a single-channel wrist EMG sensor. A dual-branch transformer with cross-modal attention simultaneously estimates 3D hand pose for 21 joints and fingertip forces for five fingers, trained with a loss that enforces kinematic realism and smooth force trajectories. Across 20 participants and 20 gestures, the system achieves MPJPE around 0.57 cm and fingertip force RMSE ~0.213 with Pearson r ~0.76, while maintaining real-time performance suitable for VR/AR and interactive applications. The work demonstrates strong generalization, ablation-supported contributions of each modality, and compelling use cases, including a Unity VR demo and daily-life force-aware interactions, highlighting potential impacts in immersive interfaces, assistive devices, and ergonomic monitoring.

Abstract

Hand pose tracking is essential for advancing applications in human-computer interaction. Current approaches, such as vision-based systems and wearable devices, face limitations in portability, usability, and practicality. We present a novel wearable system that reconstructs 3D hand pose and estimates per-finger forces using a minimal ring-watch sensor setup. A ring worn on the finger integrates an inertial measurement unit (IMU) to capture finger motion, while a smartwatch-based single-channel electromyography (EMG) sensor on the wrist detects muscle activations. By leveraging the complementary strengths of motion sensing and muscle signals, our approach achieves accurate hand pose tracking and grip force estimation in a compact wearable form factor. We develop a dual-branch transformer network that fuses IMU and EMG data with cross-modal attention to predict finger joint positions and forces simultaneously. A custom loss function imposes kinematic constraints for smooth force variation and realistic force saturation. Evaluation with 20 participants performing daily object interaction gestures demonstrates an average Mean Per Joint Position Error (MPJPE) of 0.57 cm and a fingertip force estimation (RMSE: 0.213, r=0.76). We showcase our system in a real-time Unity application, enabling virtual hand interactions that respond to user-applied forces. This minimal, force-aware tracking system has broad implications for VR/AR, assistive prosthetics, and ergonomic monitoring.

Figures (12)

• Figure 1: Ring-watch Hardware.
• Figure 2: Complementary sensing of hand grasp interaction from both physiological and motion phase.
• Figure 3: Ring-Watch Wearables. (a) Ring (b) main components of the ring prototype. (c) circuitry of the ring. (d) sensor-side of the watch prototype. (e) wireless EMG process board (front-side). (f) gold-plated EMG dry electrodes (back-side). (g) circuitry of the EMG sensor. (h) layout of EMG electrodes
• Figure 4: The architecture of the hand pose and pressure reconstruction network that integrates IMU and EMG data.
• Figure 5: Experiment setup. Ground-truth forces were measured via a five-channel fingertip force sensor.