Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

THETA: Triangulated Hand-State Estimation for Teleoperation and Automation in Robotic Hand Control

Alex Huang, Akshay Karthik

TL;DR

THETA tackles the cost and practicality barriers of hand-state tracking for teleoperation by leveraging triangulation from three RGB webcams to estimate finger joint angles in real time. The approach integrates a DeepLabV3-based hand segmentation backbone with a MobileNetV2 classifier that maps multi-view, segmented frames to discrete joint-angle bins, and streams predictions via serial to a DexHand for actuation. On a dataset of over 48,000 images spanning 40 gestures, THETA achieves about 97% accuracy with strong recall (~0.99) and robust precision, demonstrating reliable performance across varied lighting and viewpoints. The work presents a compact, low-cost teleoperation pipeline with potential extensions to regression-based joint-angle prediction and user-specific adaptation, broadening accessibility for medical, linguistic, and manufacturing applications.

Abstract

The teleoperation of robotic hands is limited by the high costs of depth cameras and sensor gloves, commonly used to estimate hand relative joint positions (XYZ). We present a novel, cost-effective approach using three webcams for triangulation-based tracking to approximate relative joint angles (theta) of human fingers. We also introduce a modified DexHand, a low-cost robotic hand from TheRobotStudio, to demonstrate THETA's real-time application. Data collection involved 40 distinct hand gestures using three 640x480p webcams arranged at 120-degree intervals, generating over 48,000 RGB images. Joint angles were manually determined by measuring midpoints of the MCP, PIP, and DIP finger joints. Captured RGB frames were processed using a DeepLabV3 segmentation model with a ResNet-50 backbone for multi-scale hand segmentation. The segmented images were then HSV-filtered and fed into THETA's architecture, consisting of a MobileNetV2-based CNN classifier optimized for hierarchical spatial feature extraction and a 9-channel input tensor encoding multi-perspective hand representations. The classification model maps segmented hand views into discrete joint angles, achieving 97.18% accuracy, 98.72% recall, F1 Score of 0.9274, and a precision of 0.8906. In real-time inference, THETA captures simultaneous frames, segments hand regions, filters them, and compiles a 9-channel tensor for classification. Joint-angle predictions are relayed via serial to an Arduino, enabling the DexHand to replicate hand movements. Future research will increase dataset diversity, integrate wrist tracking, and apply computer vision techniques such as OpenAI-Vision. THETA potentially ensures cost-effective, user-friendly teleoperation for medical, linguistic, and manufacturing applications.

THETA: Triangulated Hand-State Estimation for Teleoperation and Automation in Robotic Hand Control

TL;DR

THETA tackles the cost and practicality barriers of hand-state tracking for teleoperation by leveraging triangulation from three RGB webcams to estimate finger joint angles in real time. The approach integrates a DeepLabV3-based hand segmentation backbone with a MobileNetV2 classifier that maps multi-view, segmented frames to discrete joint-angle bins, and streams predictions via serial to a DexHand for actuation. On a dataset of over 48,000 images spanning 40 gestures, THETA achieves about 97% accuracy with strong recall (~0.99) and robust precision, demonstrating reliable performance across varied lighting and viewpoints. The work presents a compact, low-cost teleoperation pipeline with potential extensions to regression-based joint-angle prediction and user-specific adaptation, broadening accessibility for medical, linguistic, and manufacturing applications.

Abstract

The teleoperation of robotic hands is limited by the high costs of depth cameras and sensor gloves, commonly used to estimate hand relative joint positions (XYZ). We present a novel, cost-effective approach using three webcams for triangulation-based tracking to approximate relative joint angles (theta) of human fingers. We also introduce a modified DexHand, a low-cost robotic hand from TheRobotStudio, to demonstrate THETA's real-time application. Data collection involved 40 distinct hand gestures using three 640x480p webcams arranged at 120-degree intervals, generating over 48,000 RGB images. Joint angles were manually determined by measuring midpoints of the MCP, PIP, and DIP finger joints. Captured RGB frames were processed using a DeepLabV3 segmentation model with a ResNet-50 backbone for multi-scale hand segmentation. The segmented images were then HSV-filtered and fed into THETA's architecture, consisting of a MobileNetV2-based CNN classifier optimized for hierarchical spatial feature extraction and a 9-channel input tensor encoding multi-perspective hand representations. The classification model maps segmented hand views into discrete joint angles, achieving 97.18% accuracy, 98.72% recall, F1 Score of 0.9274, and a precision of 0.8906. In real-time inference, THETA captures simultaneous frames, segments hand regions, filters them, and compiles a 9-channel tensor for classification. Joint-angle predictions are relayed via serial to an Arduino, enabling the DexHand to replicate hand movements. Future research will increase dataset diversity, integrate wrist tracking, and apply computer vision techniques such as OpenAI-Vision. THETA potentially ensures cost-effective, user-friendly teleoperation for medical, linguistic, and manufacturing applications.
Paper Structure (11 sections, 9 figures, 3 tables)

This paper contains 11 sections, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Depth-Sensing & Infrared Systems
  4. Sensor Glove-Based Systems
  5. Vision-Based Estimation with RGB Cameras
  6. Methodology
  7. DexHand Robotic Hand Design & ROS 2 Control
  8. THETA Architectural Pipeline Multi-View Data Collection, Annotation & Segmentation
  9. THETA Architecture Pipeline Segmentation Preprocessing & Joint Angle Classification
  10. Results & Analysis
  11. Conclusions & Future Works

Figures (9)

  • Figure 1: Assembled DexHand (with personal modifications)
  • Figure 2: ROS 2-Arduino Joint Angle Transmission pipeline for robotic hand servos actuation.
  • Figure 3: Triangulation Data Collection Setup
  • Figure 4: Multi-View RGB Image Segmentation Using DeepLabV3 for Image Preprocessing, Feature Extraction, Segmentation Prediction, and Mask Generation.
  • Figure 5: Multi-View RGB Image Segmentation Using DeepLabV3 for Image Preprocessing, Feature Extraction, Segmentation Prediction, and Mask Generation.
  • ...and 4 more figures