Radiance Fields for Robotic Teleoperation

TL;DR

This work introduces an online Radiance Field pipeline for robotic teleoperation that unifies multi-camera data, NeRF and 3D Gaussian Splatting (3DGS) reconstructions, and immersive visualization. By integrating NerfStudio within a ROS-friendly Radiance Field Node and providing both RViz and VR interfaces, the approach delivers high-fidelity, maneuverable scene representations during teleoperation. Comparative experiments across static arms, mobile bases, and mobile arms show radiance-field methods generally outperform mesh baselines, with 3DGS delivering real-time rendering and NeRF providing strong perceptual quality, while VR visualization enhances operator usability and depth perception. The results indicate a practical path toward VR-ready, online radiance-field teleoperation, with evidence favoring explicit representations for perception and manipulation and highlighting future work in fully pushing 3DGS into immersive workflows.

Abstract

Radiance field methods such as Neural Radiance Fields (NeRFs) or 3D Gaussian Splatting (3DGS), have revolutionized graphics and novel view synthesis. Their ability to synthesize new viewpoints with photo-realistic quality, as well as capture complex volumetric and specular scenes, makes them an ideal visualization for robotic teleoperation setups. Direct camera teleoperation provides high-fidelity operation at the cost of maneuverability, while reconstruction-based approaches offer controllable scenes with lower fidelity. With this in mind, we propose replacing the traditional reconstruction-visualization components of the robotic teleoperation pipeline with online Radiance Fields, offering highly maneuverable scenes with photorealistic quality. As such, there are three main contributions to state of the art: (1) online training of Radiance Fields using live data from multiple cameras, (2) support for a variety of radiance methods including NeRF and 3DGS, (3) visualization suite for these methods including a virtual reality scene. To enable seamless integration with existing setups, these components were tested with multiple robots in multiple configurations and were displayed using traditional tools as well as the VR headset. The results across methods and robots were compared quantitatively to a baseline of mesh reconstruction, and a user study was conducted to compare the different visualization methods. For videos and code, check out https://rffr.leggedrobotics.com/works/teleoperation/.

Figures (10)

• Figure 1: Teleoperator controlling a robot using a VR interface from inside a reconstructed Radiance Field. Reconstructions are created online from any robot based on its sensor configuration. Neural Radiance Field or Gaussian Splatting renders can be displayed in an immersive 360$^{\circ}$ render or on a handheld viewer. These renders are displayed alongside the robot and its sensor data, such as camera feeds and LiDAR.
• Figure 2: General teleoperation visualization pipeline divided into three sections: Robots, Reconstruction Methods, Visualization. Sensor and pose data flows from various robotic components (red) into the reconstruction method (green) to create a scene representation that is shown to the user in the visualizer (blue). Support for Radiance Field reconstructions such as Neural Radiance Fields (NeRFs) and Gaussian Splatting, as well as RViz and VR visualizers for these methods, are presented in this work. For baseline comparisons, a Voxblox mesh viewer was also ported to VR. These contributions are highlighted with a purple dashed line.
• Figure 3: Component diagram showing the connections between the robotic systems (red and orange), the reconstruction methods (green), and the visualizers (blue). Data flows from the robotic systems into reconstruction method nodes where it is either merged into rendered views for Radiance Fields or a mesh for Voxblox. The Radiance Field Node is comprised of a custom DataLoader, DataParser, and Dataset, and uses a collection of Sensor objects to manage the ROS subscribers. The DataLoader and Dataset send data to any standard method during training, only returning images and cameras that have been updated. This data is then displayed directly in RViz or transmitted over TCP to a VR headset.
• Figure 4: Sample of the RViz plugin occluding the robot based on scene depth (left) and having the robot rendered on top (right).
• Figure 5: An axis-aligned bounding box can be used to crop the background (left) or remove walls (right) allowing for novel views and clearer operation.