Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Soft Vision-Based Tactile-Enabled SixthFinger: Advancing Daily Objects Manipulation for Stroke Survivors

Basma Hasanen, Mashood M. Mohsan, Abdulaziz Y. Alkayas, Federico Renda, Irfan Hussain

TL;DR

The paper addresses post-stroke grasping deficits by introducing a soft, vision-based tactile-enabled extra finger (VTE-SF) that autonomously modulates grip in response to slip. It combines GelSight-based tactile sensing with lightweight vision transformers (MobileViT for touch, TimeSformer for slip) to detect contact and slip and to adjust force without imposing cognitive load on the user. Key contributions include the VTE-SF design, a transformer-based grasping framework, ablation studies, and demonstrations on daily objects showing high grasp-success and slip-robustness. The work suggests meaningful potential for daily-life assistance and rehabilitation, with future plans for user studies and enhanced haptic feedback.

Abstract

The presence of post-stroke grasping deficiencies highlights the critical need for the development and implementation of advanced compensatory strategies. This paper introduces a novel system to aid chronic stroke survivors through the development of a soft, vision-based, tactile-enabled extra robotic finger. By incorporating vision-based tactile sensing, the system autonomously adjusts grip force in response to slippage detection. This synergy not only ensures mechanical stability but also enriches tactile feedback, mimicking the dynamics of human-object interactions. At the core of our approach is a transformer-based framework trained on a comprehensive tactile dataset encompassing objects with a wide range of morphological properties, including variations in shape, size, weight, texture, and hardness. Furthermore, we validated the system's robustness in real-world applications, where it successfully manipulated various everyday objects. The promising results highlight the potential of this approach to improve the quality of life for stroke survivors.

Soft Vision-Based Tactile-Enabled SixthFinger: Advancing Daily Objects Manipulation for Stroke Survivors

TL;DR

The paper addresses post-stroke grasping deficits by introducing a soft, vision-based tactile-enabled extra finger (VTE-SF) that autonomously modulates grip in response to slip. It combines GelSight-based tactile sensing with lightweight vision transformers (MobileViT for touch, TimeSformer for slip) to detect contact and slip and to adjust force without imposing cognitive load on the user. Key contributions include the VTE-SF design, a transformer-based grasping framework, ablation studies, and demonstrations on daily objects showing high grasp-success and slip-robustness. The work suggests meaningful potential for daily-life assistance and rehabilitation, with future plans for user studies and enhanced haptic feedback.

Abstract

The presence of post-stroke grasping deficiencies highlights the critical need for the development and implementation of advanced compensatory strategies. This paper introduces a novel system to aid chronic stroke survivors through the development of a soft, vision-based, tactile-enabled extra robotic finger. By incorporating vision-based tactile sensing, the system autonomously adjusts grip force in response to slippage detection. This synergy not only ensures mechanical stability but also enriches tactile feedback, mimicking the dynamics of human-object interactions. At the core of our approach is a transformer-based framework trained on a comprehensive tactile dataset encompassing objects with a wide range of morphological properties, including variations in shape, size, weight, texture, and hardness. Furthermore, we validated the system's robustness in real-world applications, where it successfully manipulated various everyday objects. The promising results highlight the potential of this approach to improve the quality of life for stroke survivors.
Paper Structure (21 sections, 3 equations, 9 figures, 3 tables)

This paper contains 21 sections, 3 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Gelsight Vision-based Tactile Sensor
  4. Deep learning for slippage detection in grasping and tactile perception
  5. Solution Overview
  6. Proposed Approach
  7. Design and Fabrication
  8. The Vision-based Tactile Enabled SixthFinger (VTE-SF)
  9. Vision Transformers for touch and slip detection
  10. MobileViT for touch detection
  11. TimeSformer for slip detection
  12. Dataset and Data Collection
  13. Ablation Studies
  14. AB-1 (Hidden sizes)
  15. AB-2 (Attention heads)
  16. ...and 6 more sections

Figures (9)

  • Figure 1: (a) The VTE-SF system assists the stroke survivor to grasp an object (an apple) (b) The CAD design of VTE-SF. (c) The image captured by GelSight mini when the device touches the object. (d) The Sequence of frames captured by Gelsight mini when slippage of the object occurs.
  • Figure 2: The figure depicts a grasping failure scenario during liquid pouring when both functional and non-functional hands are used and force is not auto-regulated.
  • Figure 3: Proposed System Overview: 1) The VTE-SF system supporting object grasping for patient. 2) The CAD model of the proposed device. 3) System Operation Sequence: Patient positions hand near the object, initiating device motion $\rightarrow$ Touch model detects device-object contact $\rightarrow$ On touch, slip model monitors for slippage $\rightarrow$ Grasping force auto-adjusts if slippage occurs $\rightarrow$ After task completion, patient commands finger release. 4) The dimensions of the soft joint are labeled: $d_h$, and $d_t$ are the height and thickness, respectively.
  • Figure 4: Results from the parametric study conducted using the SoRoSim MATLAB toolbox: a) The 3D surface, b) The x, y projection: $d_h$, and $d_t$ are the height and thickness of the soft joints, respectively.
  • Figure 5: Transformers architecture
  • ...and 4 more figures