Low-Cost Teleoperation with Haptic Feedback through Vision-based Tactile Sensors for Rigid and Soft Object Manipulation

TL;DR

Addresses the need for tactile-informed teleoperation of delicate objects using low-cost hardware. Introduces the T2H framework that translates camera-based tactile data into vibrotactile feedback on a consumer controller and adds partial autonomy to prevent slippage, enabling safer manipulation. The haptic mapping follows a logarithmic relation $f_k = \log_{10}(1 + \alpha p_k) / \log_{10}(1 + \alpha)$ where $p_k$ is the variation metric, and integrates with a ROS–Unity bridge across two DIGIT sensors and an Oculus Quest 2. Demonstrations cover nine objects with both experienced and novice operators, showing reduced slippage and compression when partial autonomy is enabled. The code and setup are publicly released to support reproducibility and broader adoption.

Abstract

Haptic feedback is essential for humans to successfully perform complex and delicate manipulation tasks. A recent rise in tactile sensors has enabled robots to leverage the sense of touch and expand their capability drastically. However, many tasks still need human intervention/guidance. For this reason, we present a teleoperation framework designed to provide haptic feedback to human operators based on the data from camera-based tactile sensors mounted on the robot gripper. Partial autonomy is introduced to prevent slippage of grasped objects during task execution. Notably, we rely exclusively on low-cost off-the-shelf hardware to realize an affordable solution. We demonstrate the versatility of the framework on nine different objects ranging from rigid to soft and fragile ones, using three different operators on real hardware.

Figures (8)

• Figure 1: Overview of the proposed T2H teleoperation framework: the data obtained with camera-based tactile sensors are elaborated by the T2H algorithm, the resulting haptic feedback is provided to the operator via the teleoperation controller, and the commands for the robot are generated.
• Figure 2: Communication architecture for the T2H teleoperation framework.
• Figure 3: Representation of the tactile images (first row) with respective variation images (second row) as the grasp is tightened on a pistachio nut. The scene and the background are shown on the left.
• Figure 4: Representation of the haptic feedback with different gains $\alpha$.
• Figure 5: Example of the slippage reference image (left), tactile data (middle), and respective variation image (right) during a pistachio nut manipulation.