Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewards

Fatemeh Zargarbashi, Jin Cheng, Dongho Kang, Robert Sumner, Stelian Coros

TL;DR

A novel learning-based control framework that uses keyframing to incorporate high-level objectives in natural locomotion for legged robots that significantly reduces the effort of hyperparameter tuning compared to the standard single-critic alternative.

Abstract

This paper presents a novel learning-based control framework that uses keyframing to incorporate high-level objectives in natural locomotion for legged robots. These high-level objectives are specified as a variable number of partial or complete pose targets that are spaced arbitrarily in time. Our proposed framework utilizes a multi-critic reinforcement learning algorithm to effectively handle the mixture of dense and sparse rewards. Additionally, it employs a transformer-based encoder to accommodate a variable number of input targets, each associated with specific time-to-arrivals. Throughout simulation and hardware experiments, we demonstrate that our framework can effectively satisfy the target keyframe sequence at the required times. In the experiments, the multi-critic method significantly reduces the effort of hyperparameter tuning compared to the standard single-critic alternative. Moreover, the proposed transformer-based architecture enables robots to anticipate future goals, which results in quantitative improvements in their ability to reach their targets.

RobotKeyframing: Learning Locomotion with High-Level Objectives via Mixture of Dense and Sparse Rewards

TL;DR

A novel learning-based control framework that uses keyframing to incorporate high-level objectives in natural locomotion for legged robots that significantly reduces the effort of hyperparameter tuning compared to the standard single-critic alternative.

Abstract

This paper presents a novel learning-based control framework that uses keyframing to incorporate high-level objectives in natural locomotion for legged robots. These high-level objectives are specified as a variable number of partial or complete pose targets that are spaced arbitrarily in time. Our proposed framework utilizes a multi-critic reinforcement learning algorithm to effectively handle the mixture of dense and sparse rewards. Additionally, it employs a transformer-based encoder to accommodate a variable number of input targets, each associated with specific time-to-arrivals. Throughout simulation and hardware experiments, we demonstrate that our framework can effectively satisfy the target keyframe sequence at the required times. In the experiments, the multi-critic method significantly reduces the effort of hyperparameter tuning compared to the standard single-critic alternative. Moreover, the proposed transformer-based architecture enables robots to anticipate future goals, which results in quantitative improvements in their ability to reach their targets.
Paper Structure (23 sections, 11 equations, 8 figures, 9 tables, 1 algorithm)

This paper contains 23 sections, 11 equations, 8 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning for Legged Robots
  4. Natural Motion for Characters and Robots
  5. Method
  6. Problem Setup
  7. Multi-Critic RL for Dense-Sparse Reward Mixture
  8. Transformer-based Keyframe Encoding
  9. Results
  10. Keyframe Matching
  11. Multi-Critic RL
  12. Future Goal Anticipation
  13. Hardware Deployment
  14. Discussion
  15. Appendix
  16. ...and 8 more sections

Figures (8)

  • Figure 1: RobotKeyframing: Locomotion policy trained with our framework meets the keyframes with position and full posture targets (yellow) at specified times on hardware experiments.
  • Figure 2: Multi-Critic RL.
  • Figure 3: Policy with transformer-based keyframe encoder.
  • Figure 4: a) Horizontal trajectories of the robot base given two sets of position goals (dots). b) Specifying different temporal profiles generates diverse behaviors for the same position goal.
  • Figure 5: Snapshots of the robot motion given keyframes with full postures: moving forward (top), jumping (middle) and raising the paw up (bottom). Target keyframes are displayed in yellow.
  • ...and 3 more figures