An Immersive Virtual Reality Bimanual Telerobotic System With Haptic Feedback

TL;DR

This work addresses the limitations of visual-only teleoperation by introducing an immersive, bilateral bimanual telerobotic system with grounded haptic feedback and visual rendering. The approach tightly integrates dual-arm dexterous manipulation, fingertip tactile sensing, a Dexmo glove, and VR-based telepresence, with control signals derived from operator wrist motion and isomorphic finger mapping. Experimental results show that haptic feedback improves object perception, compensates for occlusions, enhances fine manipulation, and reduces cognitive burden, leading to higher efficiency and more realistic teleoperation. The system’s real-time torque feedback ($\tau = F \cdot L$) and tactile sensing enable intuitive, immersive interaction, with clear potential for broader tasks and richer auxiliary feedback in future work.

Abstract

In robotic bimanual teleoperation, multimodal sensory feedback plays a crucial role, providing operators with a more immersive operating experience, reducing cognitive burden, and improving operating efficiency. In this study, we develop an immersive bilateral isomorphic bimanual telerobotic system, which comprises dual arm and dual dexterous hands, with visual and haptic force feedback. To assess the performance of this system, we carried out a series of experiments and investigated the user's teleoperation experience. The results demonstrate that haptic force feedback enhances physical perception capabilities and complex task operating abilities. In addition, it compensates for visual perception deficiencies and reduces the operator's work burden. Consequently, our proposed system achieves more intuitive, realistic and immersive teleoperation, improves operating efficiency, and expands the complexity of tasks that robots can perform through teleoperation.

Figures (13)

• Figure 1: The proposed system for immersive and intuitive teleoperation with bilateral haptic feedback, and visual rending, isomorphic dexterous arm and hand control.
• Figure 2: Illustration of haptic rendering. (a) Each finger of the Dexmo exoskeleton glove is equipped with a motor to provide haptic feedback. (b) The human operator senses haptic force feedback through the Dexmo exoskeleton glove. (c) A tactile sensor with 16 sense units is built into the dexterous hand. (d) Example of a haptic sensor matrix when a sensor is triggered. (e) Grasping example.
• Figure 3: Illustration of the torque calculation. The $F$ refers the force applying on fingertip (see \ref{['eq:F1']}). The $L$ is the force arm.
• Figure 4: (a) and (b), the position and orientation of the bimanual robot's arm end-effector are controlled through the relative position and orientation of operator's wrist. (c) The dexmo exoskeleton glove and VIVE Tracker are bound together, and the wrist position and orientation are read through the locator. The joint angles of the 7-DoF robotic arm are calculated using traditional inverse kinematics algorithms.
• Figure 5: (a) the bend and split angles are read by sensors built in exoskeleton glove. The info obtained by sensors will be used to control the dexterous hand for isomorphic teleoperation. (b) dynamic examples of finger bending and thumb splitting.