Helping Blind People Grasp: Enhancing a Tactile Bracelet with an Automated Hand Navigation System

TL;DR

This work presents an AI-driven automated hand navigation system (HANS) that augments a tactile bracelet to help visually impaired users grasp target objects without an external operator. By leveraging dual YOLOv5 detectors for objects and hands, a StrongSORT tracker, and monocular depth estimation, the system translates visual cues into vibration-based guidance, enabling autonomous navigation in cluttered environments and around obstacles. Across grasping, multi-object tracking, and depth-navigation tasks, the approach achieves robust object localization, high success rates (notably 75% overall with expert users), and positive subjective feedback from both blindfolded and blind participants, including a real-world cafe test. The results suggest a viable path toward independent daily use, with identified bottlenecks in speed and hardware scalability and clear directions toward software/hardware optimizations and natural language interfaces for target specification.

Abstract

Grasping constitutes a critical challenge for visually impaired people. To address this problem, we developed a tactile bracelet that assists in grasping by guiding the user's hand to a target object using vibration commands. Here we demonstrate the fully automated system around the bracelet, which can confidently detect and track target and distractor objects and reliably guide the user's hand. We validate our approach in three tasks that resemble complex, everyday use cases. In a grasping task, the participants grasp varying target objects on a table, guided via the automated hand navigation system. In the multiple objects task, participants grasp objects from the same class, demonstrating our system's ability to track one specific object without targeting surrounding distractor objects. Finally, the participants grasp one specific target object by avoiding an obstacle along the way in the depth navigation task, showcasing the potential to utilize our system's depth estimations to navigate even complex scenarios. Additionally, we demonstrate that the system can aid users in the real world by testing it in a less structured environment with a blind participant. Overall, our results demonstrate that the system, by translating the AI-processed visual inputs into a reduced data rate of actionable signals, enables autonomous behavior in everyday environments, thus potentially increasing the quality of life of visually impaired people.

Figures (7)

• Figure 1: A) The tactile bracelet. B) The camera is a lightweight device attached to experimental glasses. C) Diagram presenting the pipeline of the automated hand navigation system.
• Figure 2: Schematic representation of the grasping task (A), multiple objects task (B), and depth navigation task (C).
• Figure 3: A) Detection of hand, target, and other objects in the scene, visualized by bounding boxes. B) Grasping task trial demonstration. C) Results of the experimental validation. Orange color indicates experts, blue naïve participants. D) Detection percentage during the navigation progress for successful and failed trials. As all trials had a different number of frames, they were normalized and afterwards binned into 50-step intervals.
• Figure 4: A) The target object is tracked amongst multiple instances of the same class using an identifier. B) Multiple objects task trial demonstration. C) Results of the experimental validation. Orange color indicates experts, blue naïve participants. D) Distribution of magnitudes of a tracking jump from one target object to a new one. The red line indicates the threshold for a jump occurring from one target to another (90 pixels). The histogram bins are 10 pixels wide.
• Figure 5: A) Depth map prediction for an example scene using the depth estimator. B) Depth navigation task trial demonstration with an obstacle that blocks horizontal movement. C) Depth navigation task trial demonstration with an obstacle that does not block horizontal movement. D) Results of the experimental validation. Orange color indicates experts, blue naïve participants.