Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors

TL;DR

This work investigates the use of prior knowledge of human and animal movement to learn reusable locomotion skills for real legged robots, and demonstrates how this skill module can be used for imitation, and train controllable walking and ball dribbling policies for both the ANYmal quadruped and OP3 humanoid.

Abstract

We investigate the use of prior knowledge of human and animal movement to learn reusable locomotion skills for real legged robots. Our approach builds upon previous work on imitating human or dog Motion Capture (MoCap) data to learn a movement skill module. Once learned, this skill module can be reused for complex downstream tasks. Importantly, due to the prior imposed by the MoCap data, our approach does not require extensive reward engineering to produce sensible and natural looking behavior at the time of reuse. This makes it easy to create well-regularized, task-oriented controllers that are suitable for deployment on real robots. We demonstrate how our skill module can be used for imitation, and train controllable walking and ball dribbling policies for both the ANYmal quadruped and OP3 humanoid. These policies are then deployed on hardware via zero-shot simulation-to-reality transfer. Accompanying videos are available at https://bit.ly/robot-npmp.

Figures (9)

• Figure 1: Our approach consists of four stages: 1) First, we retarget human or dog mocap data to the ANYmal or OP3 robots. 2) Next, we train a policy to imitate the reference trajectories in simulation. This policy has a hierarchical structure in which a tracking policy encodes the desired reference trajectory into a latent action that subsequentially instructs a proprioception-conditioned low-level controller. 3) We can now reuse the low-level controller by training a new task policy to output latent actions to instruct the low-level controller whose parameters are kept fixed. This enables us to solve challenging tasks such as ball dribbling. 4) Finally, we transfer our resulting controllers from simulation to real hardware in zero-shot fashion. This is realized by the use of accurate simulation models as well as dynamics and domain randomization in simulation.
• Figure 2: Result highlights. (A) Imitation of dog mocap by ANYmal. The top row shows a visualization of the original reference, the middle row imitation in simulation and the bottom row imitation in reality. (B) Top-down view of ANYmal following a slalom trajectory using a trajectory-following controller on top of a controllable-walking policy. The color gradient (light to dark) indicates position over time. Arrows indicate the target velocity of the blended keyframes. The figure highlights the accuracy and consistency with which the policy follows the instructions. An obstacle (right-hand side of the figure) is introduced during the trial. This forces the controller to locally deviate from the instructed trajectory but it recovers within a half-turn. (C) Reusing the low-level controllers for controllable walking on hilly terrain with ANYmal as well as dribbling with OP3 (D) and ANYmal (E) in simulation and real, respectively.
• Figure 3: Zero-shot imitation on ANYmal. (A) Top-down view of the path traversed by the base of the robot while following a mocap reference clip, in simulation and reality. Color darkness is proportional to time. (B) Comparison of the height of the left-front foot over time for the mocap reference, the simulated robot and the real robot.
• Figure 4: Analysis of the controllable walking results. (A) Accuracy at which the ANYmal (left) and OP3 (right) controllers follow commanded forward velocities in simulation and reality. For OP3, the real-world success rate is also plotted. Commanded velocities are held fixed throughout each trial and values shown are mean and standard error across all trials. (B) Contact patterns of the feet for both the learned and the baseline walking controller on ANYmal as estimated on the real robot during walking at fixed velocity. (C) Corresponding pitch, roll and height measures of the base. See main text for discussion.
• Figure 5: ANYmal dribbling a ball. (A) A series of keyframes that illustrate the dribbling controller using all of its legs to interact with the ball. (B) Two views of a 3D plot of the top-down trajectory followed by the dribbling controller both in simulation (green) and on the real robot (orange). Blue markers show the position of the target over time. The vertical axis represents time. Both the orange and the green trace closely follow the target positions. Note that the targets are moved after fixed time intervals and the robot sometimes has to wait for the next target to appear.