Mixed Reality Teleoperation Assistance for Direct Control of Humanoids

TL;DR

A novel approach that utilizes mixed reality and assistive autonomy to enhance the efficiency and precision of humanoid robot teleoperation is introduced, which combines user motion with autonomous capabilities, achieving task efficiency while maintaining human-like robot motion.

Abstract

Teleoperation plays a crucial role in enabling robot operations in challenging environments, yet existing limitations in effectiveness and accuracy necessitate the development of innovative strategies for improving teleoperated tasks. This article introduces a novel approach that utilizes mixed reality and assistive autonomy to enhance the efficiency and precision of humanoid robot teleoperation. By leveraging Probabilistic Movement Primitives, object detection, and Affordance Templates, the assistance combines user motion with autonomous capabilities, achieving task efficiency while maintaining human-like robot motion. Experiments and feasibility studies on the Nadia robot confirm the effectiveness of the proposed framework.

Figures (7)

• Figure 1: With the teleoperation assistance we propose, the user serves as a high-level guide, directing the robot's autonomous task execution through a mixed reality interface (top left) and specifying the manner in which the task should be carried out. In the given example, the user merely initiates a portion of a punching motion, specifically indicating an uppercut technique. Recognizing this, the robot discerns that it is tasked with a punching action and, among the various techniques available, it should execute an uppercut. The user does not need to focus on the precision of the subsequent motion, as the initial input is sufficient for the robot to complete the task accurately.
• Figure 2: Flowchart of the assistive autonomy. During the training phase, the human operator teleoperates the robot in simulation, and performs the tasks in different ways. A ProMP is learned for every task. When teleoperating the robot in real-time, the ProMPs are used to generate the assistive robot motion: (1) the system recognizes the current task, with the help of context given by the object detection; (2) it updates the ProMP according to the initially observed user input and object pose; (3) if an affordance is available for that object, a blending mechanism is used to to achieve smooth transitions between ProMP-generated motions and ATs.
• Figure 3: Operator's view in mixed reality while remotely controlling the robot using the proposed teleoperation assistance. The virtual panel provides real-time guidance, presenting the user with essential instructions for each step of the operation. Pre-activation. An object has been detected (highlighted in blue and with an orange arrow) for which the autonomy is available, and the user can activate the teleoperation assistance. Generation. The user can start doing the task and provide initial input to the autonomy that will adapt accordingly. Validation. The user can preview the proposed motion via a ghost robot and spline trajectories, validate it or reject it. Execution. The user can execute the proposed motion via joystick control.
• Figure 4: Interdependence analysis of a teleoperated task using the assistive autonomy.
• Figure 5: ProMP-based reference trajectories for reaching the door handle. Left column: learned ProMPs (light thick lines with transparent regions) alongside the corresponding 20 demonstrations (dark thin lines), expressed in the door handle frame. These demonstrations include reaching motions from different approach locations. Right column: updated ProMPs (dark lines) after observing the user input (lighter short lines followed by a vertical gray bar).