Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating and Evaluating Visuo-tactile Sensing with Haptic Feedback for Teleoperated Robot Manipulation

Noah Becker, Kyrylo Sovailo, Chunyao Zhu, Erik Gattung, Kay Hansel, Tim Schneider, Yaonan Zhu, Yasuhisa Hasegawa, Jan Peters

TL;DR

This work tackles the limited tactile feedback in teleoperation by integrating GelSight visuotactile sensing on the robot gripper with vibrotactile feedback delivered through MANUS gloves within a VR-based dual-arm teleoperation pipeline. It presents two force-estimation methods—Lucas-Kanade optical motion flow and a neural shear-force estimator—with normalization and thresholds, enabling real-time haptic feedback. A novel bench setup is introduced to quantify normal force, shear force, and slip, facilitating controlled evaluation and data collection. Preliminary user studies indicate a substantial reduction in object deformation when haptic feedback is enabled, though users report increased control effort due to heightened sensitivity, underscoring both the promise and the challenges of visuotactile-enabled teleoperation for practical use.

Abstract

Telerobotics enables humans to overcome spatial constraints and physically interact with the environment in remote locations. However, the sensory feedback provided by the system to the user is often purely visual, limiting the user's dexterity in manipulation tasks. This work addresses this issue by equipping the robot's end-effector with high-resolution visuotactile GelSight sensors. Using low-cost MANUS-Gloves, we provide the user with haptic feedback about forces acting at the points of contact in the form of vibration signals. We employ two different methods for estimating these forces; one based on estimating the movement of markers on the sensor surface and one deep-learning approach. Additionally, we integrate our system into a virtual-reality teleoperation pipeline in which a human user controls both arms of a Tiago robot while receiving visual and haptic feedback. Lastly, we present a novel setup to evaluate normal force, shear force, and slip. We believe that integrating haptic feedback is a crucial step towards dexterous manipulation in teleoperated robotic systems.

Integrating and Evaluating Visuo-tactile Sensing with Haptic Feedback for Teleoperated Robot Manipulation

TL;DR

This work tackles the limited tactile feedback in teleoperation by integrating GelSight visuotactile sensing on the robot gripper with vibrotactile feedback delivered through MANUS gloves within a VR-based dual-arm teleoperation pipeline. It presents two force-estimation methods—Lucas-Kanade optical motion flow and a neural shear-force estimator—with normalization and thresholds, enabling real-time haptic feedback. A novel bench setup is introduced to quantify normal force, shear force, and slip, facilitating controlled evaluation and data collection. Preliminary user studies indicate a substantial reduction in object deformation when haptic feedback is enabled, though users report increased control effort due to heightened sensitivity, underscoring both the promise and the challenges of visuotactile-enabled teleoperation for practical use.

Abstract

Telerobotics enables humans to overcome spatial constraints and physically interact with the environment in remote locations. However, the sensory feedback provided by the system to the user is often purely visual, limiting the user's dexterity in manipulation tasks. This work addresses this issue by equipping the robot's end-effector with high-resolution visuotactile GelSight sensors. Using low-cost MANUS-Gloves, we provide the user with haptic feedback about forces acting at the points of contact in the form of vibration signals. We employ two different methods for estimating these forces; one based on estimating the movement of markers on the sensor surface and one deep-learning approach. Additionally, we integrate our system into a virtual-reality teleoperation pipeline in which a human user controls both arms of a Tiago robot while receiving visual and haptic feedback. Lastly, we present a novel setup to evaluate normal force, shear force, and slip. We believe that integrating haptic feedback is a crucial step towards dexterous manipulation in teleoperated robotic systems.
Paper Structure (4 sections, 2 figures)

This paper contains 4 sections, 2 figures.

Table of Contents

  1. INTRODUCTION
  2. SYSTEM AND METHOD
  3. EVALUATION
  4. Conclusion

Figures (2)

  • Figure 1: The Teleoperation setup. Left - Tiago robot equipped with tactile GelSight sensor. Top right - Grasping modeling compound, with GelSight in white finger. Bottom right - Visualisation of motion flow used for force estimation.
  • Figure 2: Overview of different components of the proposed system. (a) - The haptic feedback pipeline. (b) - The teleoperation pipeline. (c) - The test setup for determining the slip and shear forces. (c) left - Rearranged for better visibility of the individual components.(c) right - Our complete test setup including casing.