Extended Reality System for Robotic Learning from Human Demonstration

TL;DR

The Robot Action Demonstration in Extended Reality (RADER) system for learning from demonstration approaches is proposed and its application to a state-of-the-art approach is presented and comparable results between using a physical robot and the system are shown.

Abstract

Many real-world tasks are intuitive for a human to perform, but difficult to encode algorithmically when utilizing a robot to perform the tasks. In these scenarios, robotic systems can benefit from expert demonstrations to learn how to perform each task. In many settings, it may be difficult or unsafe to use a physical robot to provide these demonstrations, for example, considering cooking tasks such as slicing with a knife. Extended reality provides a natural setting for demonstrating robotic trajectories while bypassing safety concerns and providing a broader range of interaction modalities. We propose the Robot Action Demonstration in Extended Reality (RADER) system, a generic extended reality interface for learning from demonstration. We additionally present its application to an existing state-of-the-art learning from demonstration approach and show comparable results between demonstrations given on a physical robot and those given using our extended reality system.

Figures (15)

• Figure 1: Diagram of manipulator arm components. A manipulator is composed of many links connected via joints. The orange link and the joint indicated by the red arrow are the parent link and joint respectively of the green link.
• Figure 2: RADER system block diagram. As the human user interacts with the robot, collected trajectories are sent to application ROS nodes, which collect and process the data. Depending on the application, feedback may be provided by these nodes. Trajectories may be executed on a physical robot.
• Figure 3: One of the default menus provided in RADER allows the user to select a joint from a drop-down menu and precisely set its angle value. This menu also provides the user the option to start and subsequently stop recording a demonstration.
• Figure 4: The color of the mesh corresponding to a joint being hovered over or selected changes to inform the user of which joint they can interact with.
• Figure 5: When the user moves the target sphere at the end of the end effector, inverse kinematics is used to find a robot configuration which places the end effector as close to the sphere as possible.