Leveraging Tactile Sensing to Render both Haptic Feedback and Virtual Reality 3D Object Reconstruction in Robotic Telemanipulation

TL;DR

The paper tackles the challenge of camera-free robotic teleoperation by leveraging tactile sensing to reconstruct real-time 3D object shapes and render haptic feedback via VR. The approach combines a Leader–Follower teleoperation setup with Gaussian Process shape estimation to produce a real-time Signed Distance Field representation of the manipulated object, streamed to a Meta Quest 2 headset along with kinesthetic feedback. Key contributions include integrating VR visualization with tactile-based 3D reconstruction for blind manipulation and providing experimental evidence that precise pick-and-place is feasible without cameras, though performance depends on object geometry and operator fatigue. This work advances camera-free telemanipulation by demonstrating a practical pathway toward robust haptic-enabled manipulation in low-visibility environments and informs future enhancements in immersive VR and sensorization of the robotic hand.

Abstract

Dexterous robotic manipulator teleoperation is widely used in many applications, either where it is convenient to keep the human inside the control loop, or to train advanced robot agents. So far, this technology has been used in combination with camera systems with remarkable success. On the other hand, only a limited number of studies have focused on leveraging haptic feedback from tactile sensors in contexts where camera-based systems fail, such as due to self-occlusions or poor light conditions like smoke. This study demonstrates the feasibility of precise pick-and-place teleoperation without cameras by leveraging tactile-based 3D object reconstruction in VR and providing haptic feedback to a blindfolded user. Our preliminary results show that integrating these technologies enables the successful completion of telemanipulation tasks previously dependent on cameras, paving the way for more complex future applications.

Figures (5)

• Figure 1: View of the human-operator setup (haptic interface with the glove and a VR visor) and the teleoperated robotic hand.
• Figure 2: Control Scheme for Blind Bilateral Teleoperation : The operator controls the UR5 end effector using Virtuose 6D for pose and HGlove for Allegro Hand fingers. A haptic virtual fixture restricts movement beyond the blue box. Custom tactile sensors on the robotic hand provide kinesthetic feedback and real-time shape reconstruction. Visualization meshes streamed via Meta Quest 2 VR visor improve teleoperation transparency.
• Figure 3: Virtual Reality Scene: This series of images captures various stages of an experiment from the operator's viewpoint, recorded using the Meta Quest 2. On the grey worktable, a red semi-sphere renders the GP prior, which dynamically shifts to blue following contact measurements to approximate the shape of the manipulated object. The timer depicted is illustrative and does not correspond to an actual experiment.
• Figure 4: Pick and Place Results: Each object is distinctly coloured, and the virtual representation of the target's base is depicted with a black dashed line. The contours of the object, placed after each trial, are illustrated using five different colours. Failure of the task, meaning the object is not placed in its original O1, O2, or O3 configuration at the end of the trials, results in an omission of the contour representation on the graph.
• Figure 5: Pick and Place Errors. Each of the three bar charts displays, for a specific object, the position error (d) in blue (mean and standard deviation of the five repetitions), the orientation error ($\alpha$) in green (mean and standard deviation of the five repetitions), and the number of failed attempts in red, for each experimental Session. The image on the bottom right corner explains visually our definition of position error (d) and orientation error ($\alpha$).