Augmented Reality Demonstrations for Scalable Robot Imitation Learning

TL;DR

This paper presents an innovative solution: an Augmented Reality (AR)-assisted framework for demonstration collection, empowering non-roboticist users to produce demonstrations for robot IL using devices like the HoloLens 2.0.

Abstract

Robot Imitation Learning (IL) is a widely used method for training robots to perform manipulation tasks that involve mimicking human demonstrations to acquire skills. However, its practicality has been limited due to its requirement that users be trained in operating real robot arms to provide demonstrations. This paper presents an innovative solution: an Augmented Reality (AR)-assisted framework for demonstration collection, empowering non-roboticist users to produce demonstrations for robot IL using devices like the HoloLens 2. Our framework facilitates scalable and diverse demonstration collection for real-world tasks. We validate our approach with experiments on three classical robotics tasks: reach, push, and pick-and-place. The real robot performs each task successfully while replaying demonstrations collected via AR.

Figures (4)

• Figure 1: This figure illustrates the proposed framework's pipeline. The user with AR glasses performs the task while being mindful of the attached AR robot end effector. We capture the low-dimensional states as demonstrations during this process. The collected demonstrations can be readily applied to various downstream tasks, such as demonstration replay and training imitation learning algorithms.
• Figure 2: Left: Egocentric view showing the overlap between the human hand and the robot end effector. Right: The human performs the task manually, with the robot end effector mirroring the hand movements.
• Figure 3: Each column represents a distinct task. In the upper figure of each task, the visualization displays the demonstration without processing, with red points indicating detected key data points. The lower figure illustrates the demonstration after processing. In all tasks, the blue triangle denotes the initial position of the robot arm's end-effector. For the Reach task, green triangles mark the three-goal waypoints. In the Push and Pick-And-Place tasks, green triangles indicate the starting and goal positions for the object.
• Figure 4: Each row represents a specific task: Reach, Push, and Pick-And-Place, respectively.