Vision Controlled Orthotic Hand Exoskeleton
Connor Blais, Md Abdul Baset Sarker, Masudul H. Imtiaz
TL;DR
This work addresses the challenge of enabling calibration-free control for a wearable hand exoskeleton using AI vision. The authors implement a vision-based control loop on a Google Coral Dev Board Micro, evaluating MobileNet_V2 and YOLOv11 on the Edge TPU to achieve real-time object detection and pneumatic grasping. The MobileNet_V2 approach achieves about 10 FPS with a moderate $AP$ of $0.67$, enabling an 8+ hour runtime on a compact wrist-worn device, while YOLOv11 faces quantization obstacles. Overall, the study demonstrates the feasibility of practical, portable, vision-driven assistance for hand rehabilitation, and outlines concrete improvements needed for robust real-world deployment.
Abstract
This paper presents the design and implementation of an AI vision-controlled orthotic hand exoskeleton to enhance rehabilitation and assistive functionality for individuals with hand mobility impairments. The system leverages a Google Coral Dev Board Micro with an Edge TPU to enable real-time object detection using a customized MobileNet\_V2 model trained on a six-class dataset. The exoskeleton autonomously detects objects, estimates proximity, and triggers pneumatic actuation for grasp-and-release tasks, eliminating the need for user-specific calibration needed in traditional EMG-based systems. The design prioritizes compactness, featuring an internal battery. It achieves an 8-hour runtime with a 1300 mAh battery. Experimental results demonstrate a 51ms inference speed, a significant improvement over prior iterations, though challenges persist in model robustness under varying lighting conditions and object orientations. While the most recent YOLO model (YOLOv11) showed potential with 15.4 FPS performance, quantization issues hindered deployment. The prototype underscores the viability of vision-controlled exoskeletons for real-world assistive applications, balancing portability, efficiency, and real-time responsiveness, while highlighting future directions for model optimization and hardware miniaturization.