In-Hand Following of Deformable Linear Objects Using Dexterous Fingers with Tactile Sensing

TL;DR

This work addresses in-hand following of deformable linear objects (DLOs) using a generic dexterous hand with tactile sensing to overcome limitations of rigid grasping and parallel-gripper approaches. It proposes a unified framework combining Cartesian-space arm-hand control, tactile-based in-hand 3-D DLO pose estimation, and task-specific motion design, implemented on an open-source LEAP Hand with GelSight Mini sensors. The core contributions include an optimization-based IK solver for multi-objective fingertip control, a hybrid position/force control scheme for stable gripping, and an adaptive tactile pipeline that estimates the in-hand DLO pose via a 3-D line model parameterized as $\bm{\xi}+t\bm{\psi}$ with parameters $\Theta=(\bm{\xi},\bm{\psi},r,\Delta\bm{p})$. Experimental results show superior robustness and generalization over parallel grippers across multiple DLOs and speeds, as well as successful demonstrations of switching between grasping and following for practical tasks. The findings highlight the potential of dexterous hands with tactile sensing for real-world DLO manipulation, enabling applications such as cable routing and complex in-hand operations.”

Abstract

Most research on deformable linear object (DLO) manipulation assumes rigid grasping. However, beyond rigid grasping and re-grasping, in-hand following is also an essential skill that humans use to dexterously manipulate DLOs, which requires continuously changing the grasp point by in-hand sliding while holding the DLO to prevent it from falling. Achieving such a skill is very challenging for robots without using specially designed but not versatile end-effectors. Previous works have attempted using generic parallel grippers, but their robustness is unsatisfactory owing to the conflict between following and holding, which is hard to balance with a one-degree-of-freedom gripper. In this work, inspired by how humans use fingers to follow DLOs, we explore the usage of a generic dexterous hand with tactile sensing to imitate human skills and achieve robust in-hand DLO following. To enable the hardware system to function in the real world, we develop a framework that includes Cartesian-space arm-hand control, tactile-based in-hand 3-D DLO pose estimation, and task-specific motion design. Experimental results demonstrate the significant superiority of our method over using parallel grippers, as well as its great robustness, generalizability, and efficiency.

Figures (10)

• Figure 1: Task of in-hand DLO following. The goal is to slide along the DLO towards the other end while holding the DLO. We propose a control and sensing framework to pioneeringly achieve it using a generic dexterous hand with tactile sensing, which exhibits significantly better robustness and generalizability than existing works based on parallel grippers.
• Figure 2: Overview of this work's inspiration, hardware setup, and algorithm framework. (a) How humans use fingers to follow DLOs. Humans usually form the two fingers as a V-shape during DLO following; otherwise, the DLO will easily fall off owing to gravity, even if manipulated by humans. (b) The hardware setup and the defined frames. The thumb-tip frame is symmetrical to the index-fingertip frame. For each frame, the red, green, and blue arrow refers to the X, Y, and Z axis, respectively. (c) Our algorithm framework to enable DLO following by the dexterous hand, which includes generic control, sensing, and task-specific motion design.
• Figure 3: Tactile sensing of the in-hand DLO pose. (a) Estimation pipeline. (b) Estimation results of the in-hand DLO that rotates along the Z and Y axes of the index fingertip frame. (c) Comparison between our optimization-based 3-D line fitting method and the 3-D PCA method. The blue and red points represent the contact points on the index and thumb GelSight, respectively; the black line represents the estimated DLO axis.
• Figure 4: Superiority of V-shape fingers over parallel fingers. In the parallel setting, the contact force $F_{\rm contact}$ can only equal the gripping force $F_{\rm grip}$. In contrast, these two can be separately controlled by V-shape fingers.
• Figure 5: Orientation adjustment to better match the in-hand DLO pose. (a) Along the X-axis of the hand frame. (b) Along the Z-axis of the hand frame.