Shear-based Grasp Control for Multi-fingered Underactuated Tactile Robotic Hands

TL;DR

This paper tackles the challenge of manipulating delicate objects with an underactuated, biomimetic hand by leveraging high-resolution shear-sensitive tactile sensing. It introduces microTac sensors integrated on the Pisa/IIT SoftHand, a parallel multi-sensor processing architecture, and deep-learning models to predict 3D contact pose and forces across all fingertips. A two-tier control framework combines a SSIM-based gentle grasp with a force-feedback loop to maintain stable grasps under disturbances, enabling dynamic tasks such as pouring and tactile-led leader-follower interaction. The experimental results demonstrate stable grasping, adaptive manipulation during mass redistribution, and human-guided motion, highlighting the potential of tactile-driven underactuated hands for dexterous manipulation in unstructured settings.

Abstract

This paper presents a shear-based control scheme for grasping and manipulating delicate objects with a Pisa/IIT anthropomorphic SoftHand equipped with soft biomimetic tactile sensors on all five fingertips. These `microTac' tactile sensors are miniature versions of the TacTip vision-based tactile sensor, and can extract precise contact geometry and force information at each fingertip for use as feedback into a controller to modulate the grasp while a held object is manipulated. Using a parallel processing pipeline, we asynchronously capture tactile images and predict contact pose and force from multiple tactile sensors. Consistent pose and force models across all sensors are developed using supervised deep learning with transfer learning techniques. We then develop a grasp control framework that uses contact force feedback from all fingertip sensors simultaneously, allowing the hand to safely handle delicate objects even under external disturbances. This control framework is applied to several grasp-manipulation experiments: first, retaining a flexible cup in a grasp without crushing it under changes in object weight; second, a pouring task where the center of mass of the cup changes dynamically; and third, a tactile-driven leader-follower task where a human guides a held object. These manipulation tasks demonstrate more human-like dexterity with underactuated robotic hands by using fast reflexive control from tactile sensing.

Figures (9)

• Figure 1: CAD model of the microTac soft biomimetic optical tactile sensor. (Top) The microTac dome compared to a standard TacTip dome. (Bottom) Side and cross-sectional views of the microTac, showing the internal camera viewing the marker-pin array protruding into the fingerprint bumps. Overall, the microTac is approximately the same size as the pad of a human fingertip.
• Figure 2: Robot hardware and computing infrastructure. (a) The microTac tactile fingertips mounted on the Pisa/IIT SoftHand, which is mounted on the wrist of a UR5 robot arm. The arm-hand system is shown oriented in the neutral pose in which the palm faces horizontally with the thumb up. Also shown is the sensor frame, illustrating how pose and force variables are oriented relative to the sensing surface. (b) The system architecture of the sensing and computing components of the robotic system, comprising tactile sensor inputs to a Jetson Nano array, coupled by a router to a control PC which controls the UR5 robot arm and Pisa/IIT SoftHand. The Jetson Nano array allows on-board tactile image capture, processing and model prediction, to minimize the computational load on the control PC. (c) A visualization of the image processing pipeline, illustrating how skin deformation from sensor contact is imaged, processed and converted to pose and force values.
• Figure 3: Different approaches for supervised learning of tactile pose and force prediction models based on convolutional neural networks. 1) Individual models: 5 single models are trained on data from each sensor individually. 2) Aggregate model: one model is trained on data aggregated from all 5 sensors. 3) Progressive transfer model: one model is trained progressively on data from each of the 5 sensors in turn. 4) Standard transfer models: 5 individual sensor models are trained on data from each sensor individually, from a pre-trained model trained on data from all 5 sensors.
• Figure 4: Shear-based grasp controller architecture. The gentle grasp controller uses an SSIM measure of contact deformation to establish a stable yet gentle grasp on an object. Meanwhile a force-feedback controller uses force predictions from the tactile sensors to modulate the grasp in response to external disturbances. These controllers feed into the hardware interface (plant) comprising the SoftHand microcontroller and tactile model prediction array that are depicted in more detail in Fig. \ref{['fig:robo_platform']}.
• Figure 5: Various frames of reference and force components used for controlling the Pisa/IIT SoftHand on the robot arm. The global frame $\{x_G,y_G,z_G\}$ is the base frame of the UR5 robot arm, with the hand in the wrist frame $\{x_w,y_w,z_w\}$ of that robot. The force predictions from the individual tactile sensors are in the frames of those sensors (shown here as $\{x,y,z\}$ frames at each fingertip, where the superscript indicies P, R, M, I and T refer to the Pinky, Ring, Middle, Index and Thumb digits respectively), which are consolidated into a single force vector by averaging the shear forces at each fingertip. The $x$ and $y$ axes for the force vector are assumed to align with the $x_w$ and $y_w$ axes of the wrist frame, allowing the orientation of the shear force vector to be resolved from the robot kinematics.