Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DexterCap: An Affordable and Automated System for Capturing Dexterous Hand-Object Manipulation

Yutong Liang, Shiyi Xu, Yulong Zhang, Bowen Zhan, He Zhang, Libin Liu

TL;DR

DexterCap presents a low-cost, marker-assisted optical system for high-fidelity capture of dexterous hand–object manipulation under heavy occlusion, combining dense hand/object marker patches with an automated MANO-based reconstruction pipeline. It introduces DexterHand, a open-source HOI dataset featuring fine-grained in-hand manipulation and articulated objects (including Rubik's Cube), along with a robust detection/identification stack (CornerNet, EdgeNet, BlockNet) and a calibration-driven hand/marker solver. Quantitative results show competitive MSNR and low jerk compared with commercial marker-based systems and clear advantages over vision-only baselines, while achieving detailed hand and object kinematics at practical costs (under $6k hardware). The work enables scalable, automated data collection for dexterous manipulation research and provides a foundation for future physics-based, sim-to-real, and robotics applications, with open-source data and code to accelerate progress. It also highlights directions to further mitigate occlusion and expand the dataset with richer semantic annotations and more complex manipulations.

Abstract

Capturing fine-grained hand-object interactions is challenging due to severe self-occlusion from closely spaced fingers and the subtlety of in-hand manipulation motions. Existing optical motion capture systems rely on expensive camera setups and extensive manual post-processing, while low-cost vision-based methods often suffer from reduced accuracy and reliability under occlusion. To address these challenges, we present DexterCap, a low-cost optical capture system for dexterous in-hand manipulation. DexterCap uses dense, character-coded marker patches to achieve robust tracking under severe self-occlusion, together with an automated reconstruction pipeline that requires minimal manual effort. With DexterCap, we introduce DexterHand, a dataset of fine-grained hand-object interactions covering diverse manipulation behaviors and objects, from simple primitives to complex articulated objects such as a Rubik's Cube. We release the dataset and code to support future research on dexterous hand-object interaction.

DexterCap: An Affordable and Automated System for Capturing Dexterous Hand-Object Manipulation

TL;DR

DexterCap presents a low-cost, marker-assisted optical system for high-fidelity capture of dexterous hand–object manipulation under heavy occlusion, combining dense hand/object marker patches with an automated MANO-based reconstruction pipeline. It introduces DexterHand, a open-source HOI dataset featuring fine-grained in-hand manipulation and articulated objects (including Rubik's Cube), along with a robust detection/identification stack (CornerNet, EdgeNet, BlockNet) and a calibration-driven hand/marker solver. Quantitative results show competitive MSNR and low jerk compared with commercial marker-based systems and clear advantages over vision-only baselines, while achieving detailed hand and object kinematics at practical costs (under $6k hardware). The work enables scalable, automated data collection for dexterous manipulation research and provides a foundation for future physics-based, sim-to-real, and robotics applications, with open-source data and code to accelerate progress. It also highlights directions to further mitigate occlusion and expand the dataset with richer semantic annotations and more complex manipulations.

Abstract

Capturing fine-grained hand-object interactions is challenging due to severe self-occlusion from closely spaced fingers and the subtlety of in-hand manipulation motions. Existing optical motion capture systems rely on expensive camera setups and extensive manual post-processing, while low-cost vision-based methods often suffer from reduced accuracy and reliability under occlusion. To address these challenges, we present DexterCap, a low-cost optical capture system for dexterous in-hand manipulation. DexterCap uses dense, character-coded marker patches to achieve robust tracking under severe self-occlusion, together with an automated reconstruction pipeline that requires minimal manual effort. With DexterCap, we introduce DexterHand, a dataset of fine-grained hand-object interactions covering diverse manipulation behaviors and objects, from simple primitives to complex articulated objects such as a Rubik's Cube. We release the dataset and code to support future research on dexterous hand-object interaction.
Paper Structure (37 sections, 10 equations, 6 figures, 4 tables)

This paper contains 37 sections, 10 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Motion Capture for Human-Object Interaction
  4. Hand-Object Interaction Datasets
  5. Dexterous Hand Motion Capture System
  6. Hardware Setup and Marker Design
  7. Cameras
  8. Markers
  9. Image Processing and Marker Extraction
  10. Corner Detection
  11. Edge and Block Detection
  12. Block Identification
  13. Post-Processing
  14. 3D Marker Reconstruction
  15. Hand Motion Reconstruction
  16. ...and 22 more sections

Figures (6)

  • Figure 1: Our marker system: (a) Markers are attached to rigid hand regions (phalanges, dorsum) for accurate tracking. (b) Top: Rigid object (green cube) with visual marker patches. Bottom left: A visual marker patch printed on a transfer sticker. Bottom right: Visual markers transferred onto a medical tape for secure attachment.
  • Figure 2: Image processing pipeline for marker detection. (a) Raw input image with character-coded checkerboard markers. (b) CornerNet detection results. (c) EdgeNet edge classification results. (d) BlockNet block recognition with character identifiers.
  • Figure 3: Local joint coordinate systems defined for the MANO model.
  • Figure 4: Manually defined submeshes of the MANO hand, with each color corresponding to a specific finger segment.
  • Figure 5: Hand-object interaction in the DexterHand dataset.
  • ...and 1 more figures