SubSense: VR-Haptic and Motor Feedback for Immersive Control in Subsea Telerobotics

TL;DR

This work tackles the paucity of immersive, multisensory feedback in subsea teleoperation by introducing SubSense, a VR-haptic framework that non-invasively augments a 1-DOF gripper with glove-based haptics and real-time video in a VR environment. The system integrates a BlueROV2 with a Newton Subsea gripper, a non-invasive position-feedback mechanism, and a glove-driven mapping that provides grasp status and haptic cues, all coordinated through a ROS2 Docker workflow and MAVLink control. An eight-participant Tower of Hanoi study in a controlled tank demonstrates that VR+haptic control can improve remote situational awareness and reduce detrimental interactions compared with traditional FPV interfaces, though latency and disorientation remain challenges. Overall, SubSense provides a practical, multi-modal pipeline for enhancing subsea teleoperation, with clear directions for richer tactile feedback, latency reduction, and UI enhancements to support broader underwater tasks.

Abstract

This paper investigates the integration of haptic feedback and virtual reality (VR) control interfaces to enhance teleoperation and telemanipulation of underwater ROVs (remotely operated vehicles). Traditional ROV teleoperation relies on low-resolution 2D camera feeds and lacks immersive and sensory feedback, which diminishes situational awareness in complex subsea environments. We propose SubSense -- a novel VR-Haptic framework incorporating a non-invasive feedback interface to an otherwise 1-DOF (degree of freedom) manipulator, which is paired with the teleoperator's glove to provide haptic feedback and grasp status. Additionally, our framework integrates end-to-end software for managing control inputs and displaying immersive camera views through a VR platform. We validate the system through comprehensive experiments and user studies, demonstrating its effectiveness over conventional teleoperation interfaces, particularly for delicate manipulation tasks. Our results highlight the potential of multisensory feedback in immersive virtual environments to significantly improve remote situational awareness and mission performance, offering more intuitive and accessible ROV operations in the field.

Figures (11)

• Figure 1: Snapshots from a subsea inspection scenario are shown, where the human has to make real-time teleoperation decisions based on noisy visual feedback alone. From our field trials in underwater caves, we found that limited M2H (Machine to Human) feedback with no somatosensory information is a major limitation of existing teleoperation interfaces, which we attempt to address in SubSense.
• Figure 2: A 3-disk Tower of Hanoi (ToH) setup where participants are asked to move the ordered disk from one pole to another using an intermediate pole. (Left) first subtask of the ROV: moving the topmost disc; (right) Teleoperator is using SubSense to achieve it. A sample demo of the task being completed can be viewed here: https://youtu.be/0tOXA9pts_Y.
• Figure 3: Overview of the proposed multi-modal ROV teleoperation system is shown. The operator embodies the ROV via haptic feedback and a rendered camera feed to control its motion, camera orientation, and gripper pose. The sensorimotor via a controller, head movement, and hand glove.
• Figure 4: The controller and HMD functions for ROV teletoperation by SubSense; the hand-held controller maneuvers the ROV while the HMD motions control yaw and camera tilt.
• Figure 5: The controlling glove for the right hand to actuate the ROV gripper with haptic feedback (vibration pattern) based on the force feedback of the ROV during grasping.