RAMPA: Robotic Augmented Reality for Machine Programming by DemonstrAtion

TL;DR

RAMPA tackles safe and efficient programming by demonstration for industrial robotics through an end-to-end AR framework. It integrates Unity, ROS, and XR headsets to enable in-situ data recording, trajectory visualization, real-time hand mimicry, and on-the-fly ML training (e.g., ProMPs) with direct deployment to a UR robot. Quantitative and qualitative evaluations show RAMPA reduces task completion time, maintains learning quality, and enhances usability and perceived safety compared with kinesthetic control. This approach promises safer, more accessible, and scalable robotic programming for both novices and experts in real-world environments.

Abstract

This paper introduces Robotic Augmented Reality for Machine Programming by Demonstration (RAMPA), the first ML-integrated, XR-driven end-to-end robotic system, allowing training and deployment of ML models such as ProMPs on the fly, and utilizing the capabilities of state-of-the-art and commercially available AR headsets, e.g., Meta Quest 3, to facilitate the application of Programming by Demonstration (PbD) approaches on industrial robotic arms, e.g., Universal Robots UR10. Our approach enables in-situ data recording, visualization, and fine-tuning of skill demonstrations directly within the user's physical environment. RAMPA addresses critical challenges of PbD, such as safety concerns, programming barriers, and the inefficiency of collecting demonstrations on the actual hardware. The performance of our system is evaluated against the traditional method of kinesthetic control in teaching three different robotic manipulation tasks and analyzed with quantitative metrics, measuring task performance and completion time, trajectory smoothness, system usability, user experience, and task load using standardized surveys. Our findings indicate a substantial advancement in how robotic tasks are taught and refined, promising improvements in operational safety, efficiency, and user engagement in robotic programming.

Figures (9)

• Figure 1: With Rampa, novice and expert users can demonstrate trajectories and simulate robot movements in situ using augmented reality (AR).
• Figure 2: Rampa allows users to (a) draw trajectories with their bare hand, (b) simulate and observe their effect in situ, and (c) execute them on the real robot. (d) Rampa facilitates the direct and streamlined collection of trajectory data to train ML models. Please refer to text for use cases of the system.
• Figure 3: The user (a) draws a trajectory to the hoop, (b) rewinds the trajectory and chooses to redraw it from the current way-point using the UI menu, (c) redraws the trajectory, and (d) finalizes the trajectory.
• Figure 4: Users can save and view trajectories they have recorded, then train an ML model, condition it using virtual cubes, and generate a new trajectory.
• Figure 5: Technical workflow of Rampa.