TEGA: A Tactile-Enhanced Grasping Assistant for Assistive Robotics via Sensor Fusion and Closed-Loop Haptic Feedback

TL;DR

The tactile enhanced grasping assistant (TEGA) is presented, a closed loop assistive teleoperation framework that fuses EMG based intent2force inference with visuotactile sensing mapped into real time vibrotactile feedback via a wearable haptic vest, enabling intuitive, proportional force adjustment during manipulation.

Abstract

Recent advances in teleoperation have enabled sophisticated manipulation of dexterous robotic hands, with most systems concentrating on guiding finger positions to achieve desired grasp configurations. However, while accurate finger positioning is essential, it often overlooks the equally critical task of grasp force modulation, vital for handling objects of diverse hardness, texture, and shape. This limitation poses a significant challenge for users, especially individuals with upper limb disabilities who lack natural tactile feedback and rely on indirect cues to infer appropriate force levels. To address this gap, We present the tactile enhanced grasping assistant (TEGA), a closed loop assistive teleoperation framework that fuses EMG based intent2force inference with visuotactile sensing mapped into real time vibrotactile feedback via a wearable haptic vest, enabling intuitive, proportional force adjustment during manipulation. A wearable haptic vest delivers real time tactile feedback, allowing users to dynamically refine grasp force during manipulation. User studies confirm that the system substantially improves grasp stability and task success, underscoring its potential for assistive robotic applications.

Figures (8)

• Figure 1: Overview of the Tactile-Enhanced Grasping Assistant (TEGA) system. The system integrates EMG-based force inference, visuo-tactile sensing, and haptic feedback to enable intuitive, precise, and adaptive grasping force control through real-time closed-loop interaction. EMG signals from the user’s upper arm muscles are processed to infer grasp intent, while tactile sensor data from the robotic fingertips is converted into real-time vibrotactile feedback delivered via a wearable haptic vest.
• Figure 2: Hardware devices setup. (a) robot arm-hand system with mounted DIGIT sensors and (b) human operator with haptic vest, motion trackers and EMG sensors.
• Figure 3: Visualization of CCI and EDA. The grayscale image is extracted from the raw RGB tactile image to estimate the depth map of the deformation. The green circle marks the pixel point with the highest Contact CCI, representing the location with the most concentrated pressure. The blue circle identifies the sharpest pressure point, encompassing EDA, which represents the spatial extent of the deformation. Both CCI and EDA values are normalized within the range [0,1] and transmitted to the haptic vest, where they modulate the intensity of the corresponding vibration unit.
• Figure 4: Target objects to be tested. Object 1 (water bottle, rigid and heavy), Object 2 (plastic wet wipes container, medium stiffness), and Object 3 (bag of bread, soft and deformable).
• Figure 5: Experiment setup Experimental Setup for the TEGA System Evaluation. The setup consists of a 7-degree-of-freedom (7-DOF) Franka Emika Panda robotic arm with an Allegro Hand as the end-effector, equipped with DIGIT tactile sensors on each fingertip to capture real-time contact data. The human operator wears a motion tracker, three EMG sensors, and a bHaptics vest to facilitate teleoperation with haptic feedback. The operator’s forearm muscle contractions, detected by the EMG sensors, control the robotic hand’s grasping force, while the haptic vest provides vibrotactile feedback based on the Contact Concentration Index (CCI) and Effective Deformation Area (EDA) extracted from the tactile sensor data. The pick-and-place task is performed in a structured workspace.