ERUPT: An Open Toolkit for Interfacing with Robot Motion Planners in Extended Reality

TL;DR

This paper addresses the challenge of effectively planning robot motions in real-world, dynamic environments by enabling immersive interaction in extended reality. It introduces ERUPT, an open-source XR toolkit built on Unity/OpenXR and ROS2 that interfaces with MoveIt to create, modify, and visualize planning scenes and trajectories in 3D, while allowing execution of plans on physical robots. The key contributions include an XR interface for object and robot manipulation, a planning-dashboard workflow for MoveIt, and ROS2-based synchronization of planning scenes between XR and the planner, demonstrated in simulation and on real robots. The approach enhances spatial understanding, safety, and rapid prototyping of robot behaviors in shared spaces, with broad potential to accelerate robotics development and deployment.

Abstract

We propose the Extended Reality Universal Planning Toolkit (ERUPT), an extended reality (XR) system for interactive motion planning. Our system allows users to create and dynamically reconfigure environments while they plan robot paths. In immersive three-dimensional XR environments, users gain a greater spatial understanding. XR also unlocks a broader range of natural interaction capabilities, allowing users to grab and adjust objects in the environment similarly to the real world, rather than using a mouse and keyboard with the scene projected onto a two-dimensional computer screen. Our system integrates with MoveIt, a manipulation planning framework, allowing users to send motion planning requests and visualize the resulting robot paths in virtual or augmented reality. We provide a broad range of interaction modalities, allowing users to modify objects in the environment and interact with a virtual robot. Our system allows operators to visualize robot motions, ensuring desired behavior as it moves throughout the environment, without risk of collisions within a virtual space, and to then deploy planned paths on physical robots in the real world.

Figures (9)

• Figure 1: An example environment setup. A virtual robot and obstacle are set to mirror a physical environment so that an operator can prototype obstacle configurations and robot motions.
• Figure 2: An overview of the ERUPT system. Through the HMD, the user interacts with the virtual environment and robot, which are managed by the game engine. Updates to objects in the environment are synced over ROS2 between the XR representation and the MoveIt planning scene. Users send motion planning requests, which are forwarded to MoveIt, and the resulting paths are visualized to the user. After planning a satisfactory path, the user may execute it on a physical robot.
• Figure 3: Through the wrist menu (a), the user can add primitive shapes and edit the geometry of existing shapes. (b) When the user selects an object to edit, it is highlighted in a different material.
• Figure 4: An example of placing a robot in an augmented reality environment using a QR code indicator.
• Figure 5: The user may interact with the virtual robot by either (a) directly modifying the joint angles or (b) grabbing and dragging the end-effector using an attached indicator.