HandCept: A Visual-Inertial Fusion Framework for Accurate Proprioception in Dexterous Hands

TL;DR

HandCept tackles the proprioception bottleneck in dexterous hands by fusing a wrist-mounted RGB-D visual stream with a network of 9-axis IMUs through a latency-free Extended Kalman Filter. It leverages zero-shot sim2real training via a Blender-based synthetic data pipeline and the FFB6D pose estimator, reinforced by kinematic constraints to sharpen estimates, achieving joint-angle errors in the range of $2^{\circ}$ to $4^{\circ}$ with no observable drift. A compact, scalable IMU module supports serial and parallel configurations and enables a common base frame to simplify calibration. The approach offers state-of-the-art, drift-resistant proprioception with broad manipulation and human–robot interaction implications, and it provides an open-source rendering pipeline to support sim-to-real research.

Abstract

As robotics progresses toward general manipulation, dexterous hands are becoming increasingly critical. However, proprioception in dexterous hands remains a bottleneck due to limitations in volume and generality. In this work, we present HandCept, a novel visual-inertial proprioception framework designed to overcome the challenges of traditional joint angle estimation methods. HandCept addresses the difficulty of achieving accurate and robust joint angle estimation in dynamic environments where both visual and inertial measurements are prone to noise and drift. It leverages a zero-shot learning approach using a wrist-mounted RGB-D camera and 9-axis IMUs, fused in real time via a latency-free Extended Kalman Filter (EKF). Our results show that HandCept achieves joint angle estimation errors between $2^{\circ}$ and $4^{\circ}$ without observable drift, outperforming visual-only and inertial-only methods. Furthermore, we validate the stability and uniformity of the IMU system, demonstrating that a common base frame across IMUs simplifies system calibration. To support sim-to-real transfer, we also open-sourced our high-fidelity rendering pipeline, which is essential for training without real-world ground truth. This work offers a robust, generalizable solution for proprioception in dexterous hands, with significant implications for robotic manipulation and human-robot interaction.

Figures (7)

• Figure 1: (A) Overview of the HandCept implementation on the DexCo hand zhou2024dexterous. (B) and (C) show the physical DexCo hand; ArUco markers in (C) are used in experiment to obtain ground truth poses for each link.
• Figure 2: Algorithmic pipeline of HandCept. The ArUco code-based ground truth system is used solely for experimental validation. HandCept estimates the joint angles of the DexCo hand by fusing visual and inertial data streams. The pose estimates are refined with kinematic constraints before being fused using an Extended Kalman Filter (EKF). Due to the latency in visual processing, each visual-inertial update is used to correct the estimate at time $t+1$, starting from $k$ previous steps (i.e., at $t-k$). Meanwhile, IMU data provides real-time updates via the EKF.
• Figure 3: (A) Architecture of the modular IMU system. Using the I²C protocol, multiple 9-axis IMUs can be connected in series to expand the system. The I²C multiplexer supports parallel connections, enabling applications such as five-fingered hands. An Arduino collects and transmits the data to a computer via a serial interface. (B) Front and back views of the designed IMU module—one of the smallest known IMU boards, offering low cost and high accuracy for general-purpose use.
• Figure 4: (A) Comparison between high-fidelity rendered RGB-D images and real-world images. (B) Zero-shot 6D pose estimation results: the first row shows predictions on rendered images, and the second row shows predictions on real images across varying end-effector poses and hand configurations. (C) Training loss progression across four loss components. (D) Improved 6D pose estimation by enforcing kinematic constraints. Human-hand interference introduces inaccurate pose estimates, which are corrected by removing undesired rotations and translations through kinematic constrains.
• Figure 5: Illustration of visual-inertial updates within the EKF. Due to drift bias in IMU measurements, the inertial estimate diverges over time. Although visual estimation has latency, retrospective corrections can be applied to states from $k$ steps before the current time, allowing recomputation of the observation trajectory with higher accuracy.