Know your limits! Optimize the robot's behavior through self-awareness

TL;DR

A deep-learning model is introduced that anticipates the robot’s performance when imitating a given reference and ranks potential robot behaviors based on various criteria, such as fall likelihood, adherence to the reference motion, and smoothness, ensuring optimal robot behavior for any task command.

Abstract

As humanoid robots transition from labs to real-world environments, it is essential to democratize robot control for non-expert users. Recent human-robot imitation algorithms focus on following a reference human motion with high precision, but they are susceptible to the quality of the reference motion and require the human operator to simplify its movements to match the robot's capabilities. Instead, we consider that the robot should understand and adapt the reference motion to its own abilities, facilitating the operator's task. For that, we introduce a deep-learning model that anticipates the robot's performance when imitating a given reference. Then, our system can generate multiple references given a high-level task command, assign a score to each of them, and select the best reference to achieve the desired robot behavior. Our Self-AWare model (SAW) ranks potential robot behaviors based on various criteria, such as fall likelihood, adherence to the reference motion, and smoothness. We integrate advanced motion generation, robot control, and SAW in one unique system, ensuring optimal robot behavior for any task command. For instance, SAW can anticipate falls with 99.29% accuracy. For more information check our project page: https://evm7.github.io/Self-AWare

Figures (4)

• Figure 1: Optimizing robot behavior control from high-level task commands using self-awareness. Given an instruction by an operator, the robot generates multiple potential behaviors to accomplish the task, evaluates them based on its capabilities and limits, and selects the most suitable one to execute. This work introduces a motion adaptation with a Self-AWare model (SAW) to anticipate how well a robot can follow a given reference by ranking multiple potential behaviors and choosing the optimal one. For instance, in a scenario with three potential actions—walking, running, and jumping—the robot assesses each option and determines that walking is the most appropriate. Consequently, the robot walks to the person and says, "Hi." In this image, the generated references are shown by orange robots, while the robot's behavior when attempting to follow them is depicted in blue. Note that the reference motions are fixed just for visualizations, and do not consider gravity.
• Figure 2: Overview of our system for optimal robot behavior generation from high-level task commands with self-awareness. Upon a high-level instruction, given as trajectory or natural language description, we generate a human reference $\mathbf{H_{r}}$ for the robot to follow using MotionLCM dai2024motionlcm. Then, at each time horizon $t$, we assess the feasibility of the next future poses $\mathbf{R_{f}^t}$ to follow and optimize the robot behavior using the motion adapter module (shown in the gray center box). For that, MotionLCM is used first to generate potential new references with similar root trajectories and/or textual commands. Those references are retargeted to robot motions using ImitationNet, obtaining $[\mathbf{\hat{R}_{f, 1}^{t}}, \cdots, \mathbf{\hat{R}_{f, n}^{t}}]$. Our Self-AWare model (SAW, depicted in the light box on the right) ranks each edited references $\mathbf{\hat{R}_{f, i}^{t}}$ according to the robot's learned capabilities and the current robot states $\mathbf{\hat{R}_{o}^{t}}$. Thus, SAW infers a score vector $\hat{s}_i^t$ that describes how well the robot can follow a given reference $i$ at time $t$, and that can be summarized using a weighted sum to a single scalar per reference. The motion adapter then chooses the best reference and forwards it to I-CTRL 2024ictrl, which transfers this new reference to real robot commands $\mathbf{\hat{R}_{p}^{t}}$.
• Figure 3: Example of our Top-5 ranked edited motions given commanded human reference using our motion adaptation. Here, the motion in orange is the original reference $\mathbf{H_{r}}$, while in blue, we show Top-5 edited references $[\mathbf{\hat{H}_{f, 1}^{t}}, \cdots, \mathbf{\hat{H}_{f, 5}^{t}}]$ at different times. Note that, as SAW predicts a high likelihood of falling for the 'standing with one leg' for a long time, it tends to place the leg down and start 'walking', as instructed by the high-level task command.
• Figure 4: Robot behavior comparison between using self-awareness (in dark and light blue) or not (in yellow and brown). Given a textual command, we use MotionLCM dai2024motionlcm to generate a human reference, which we retarget to a JVRC-1 kinematics using ImitationNet imitationnet to obtain a reference robot motion. Then, I-CTRL 2024ictrl is used to refine this robot reference (in yellow) to conform with real-world physics (in orange). However, if our SAW model anticipates poor imitation (i.e., falls), we adapt the reference motion to prevent wrong behaviors. For that, we use MotionLCM+ImitationNet to generate various new robot reference motions (in dark blue) and SAW selects the more appropriate, which we then command to I-CTRL to optimize the robot behavior (in light blue) to follow the initial commands (described here by text). Note that $t_k$ indicates different time instants of the motion sequences.