Immersive Teleoperation Framework for Locomanipulation Tasks

TL;DR

This work tackles precise loco-manipulation in remote or hazardous environments by integrating immersive VR with Gaussian splatting to build a photorealistic, occlusion-aware representation of the scene for teleoperation. The authors propose a two-stage framework: first navigate via a conventional base-control interface, then perform manipulation in VR using a Gaussian splat of the environment aligned to the robot, enabling intuitive control and improved spatial awareness. A user study demonstrates substantial usability and efficiency gains over a baseline joystick-camera setup, with most participants preferring the VR interface. Real-world experiments across two tasks validate the framework's versatility, highlighting its potential to enhance precision manipulation in cluttered or occluded scenarios.

Abstract

Recent advancements in robotic loco-manipulation have leveraged Virtual Reality (VR) to enhance the precision and immersiveness of teleoperation systems, significantly outperforming traditional methods reliant on 2D camera feeds and joystick controls. Despite these advancements, challenges remain, particularly concerning user experience across different setups. This paper introduces a novel VR-based teleoperation framework designed for a robotic manipulator integrated onto a mobile platform. Central to our approach is the application of Gaussian splatting, a technique that abstracts the manipulable scene into a VR environment, thereby enabling more intuitive and immersive interactions. Users can navigate and manipulate within the virtual scene as if interacting with a real robot, enhancing both the engagement and efficacy of teleoperation tasks. An extensive user study validates our approach, demonstrating significant usability and efficiency improvements. Two-thirds (66%) of participants completed tasks faster, achieving an average time reduction of 43%. Additionally, 93% preferred the Gaussian Splat interface overall, with unanimous (100%) recommendations for future use, highlighting improvements in precision, responsiveness, and situational awareness. Finally, we demonstrate the effectiveness of our framework through real-world experiments in two distinct application scenarios, showcasing the practical capabilities and versatility of the Splat-based VR interface.

Figures (9)

• Figure 1: In this picture, we showcase an example of how our proposed framework is utilized. On the left, there is a virtual reality representation of the robot and the scene, reconstructed using Gaussian splattering. On the right, the actual robot, mounted on a mobile base, is depicted in the real environment during the manipulation phase
• Figure 2: This image details the structural scheme of the teleoperation framework
• Figure 3: In this image, we present an example of a scene reconstructed in virtual reality using Gaussian splatting techniques
• Figure 4: Results from the non-scaled questions in the user study.
• Figure 5: The baseline interface, displaying the camera view from the base of the manipulator on the left and the camera view from the end-effector on the right.