Immersive Rover Control and Obstacle Detection based on Extended Reality and Artificial Intelligence

TL;DR

The paper addresses safe lunar rover teleoperation in challenging terrains by integrating Extended Reality (XR) with AI-driven rock detection. It combines RGB-D sensing from a RealSense camera, YOLOv5 for 2D rock detection, RTAB-Map-based 3D mapping, and a ROS–Unity bridge to render an immersive XR environment for operator control. Experimental validation in a lunar analogue lab demonstrates reduced cognitive load and enhanced obstacle awareness with XR visualization compared to traditional 2D teleoperation. The work contributes a practical XR–AI architecture for rover control, 3D rock localization, and demonstrates the broader potential of immersive interfaces for space exploration and Earth-based scenarios like search and rescue.

Abstract

Lunar exploration has become a key focus, driving scientific and technological advances. Ongoing missions are deploying rovers to the surface of the Moon, targeting the far side and south pole. However, these terrains pose challenges, emphasizing the need for precise obstacles and resource detection to avoid mission risks. This work proposes a novel system that integrates eXtended Reality (XR) and Artificial Intelligence (AI) to teleoperate lunar rovers. It is capable of autonomously detecting rocks and recreating an immersive 3D virtual environment of the location of the robot. This system has been validated in a lunar laboratory to observe its advantages over traditional 2D-based teleoperation approaches

Figures (2)

• Figure 1: Architecture of the XR system connected to the lunar rover.
• Figure 2: Lunar rover teleoperated at the LunaLab to avoid the rock obstacles during the test.