Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotion

Yunho Kim, Hyunsik Oh, Jeonghyun Lee, Jinhyeok Choi, Gwanghyeon Ji, Moonkyu Jung, Donghoon Youm, Jemin Hwangbo

TL;DR

This work tackles the laborious reward engineering burden in reinforcement learning for legged locomotion by introducing a constrained reinforcement learning framework grounded in Constrained Markov Decision Processes (CMDPs). It combines a small set of rewards with interpretable probabilistic and average constraints, optimized via an adaptive interior-point method with a multi-head cost value function to scale to many constraints. The approach is validated across diverse legged robots, including six quadrupeds and one biped, in both simulation and zero-shot real-world deployment, demonstrating strong constraint satisfaction and robust performance with minimal reward terms and domain randomization. The results indicate substantial engineering advantages: constraints generalize across morphologies, reduce tuning complexity, and enable programmable behavior changes (e.g., gait or obstacle negotiation) without extensive reward design. Overall, the framework offers a scalable, generalizable pathway to high-performance legged locomotion with reduced engineering effort and reliable sim-to-real transfer.

Abstract

Several earlier studies have shown impressive control performance in complex robotic systems by designing the controller using a neural network and training it with model-free reinforcement learning. However, these outstanding controllers with natural motion style and high task performance are developed through extensive reward engineering, which is a highly laborious and time-consuming process of designing numerous reward terms and determining suitable reward coefficients. In this work, we propose a novel reinforcement learning framework for training neural network controllers for complex robotic systems consisting of both rewards and constraints. To let the engineers appropriately reflect their intent to constraints and handle them with minimal computation overhead, two constraint types and an efficient policy optimization algorithm are suggested. The learning framework is applied to train locomotion controllers for several legged robots with different morphology and physical attributes to traverse challenging terrains. Extensive simulation and real-world experiments demonstrate that performant controllers can be trained with significantly less reward engineering, by tuning only a single reward coefficient. Furthermore, a more straightforward and intuitive engineering process can be utilized, thanks to the interpretability and generalizability of constraints. The summary video is available at https://youtu.be/KAlm3yskhvM.

Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotion

TL;DR

This work tackles the laborious reward engineering burden in reinforcement learning for legged locomotion by introducing a constrained reinforcement learning framework grounded in Constrained Markov Decision Processes (CMDPs). It combines a small set of rewards with interpretable probabilistic and average constraints, optimized via an adaptive interior-point method with a multi-head cost value function to scale to many constraints. The approach is validated across diverse legged robots, including six quadrupeds and one biped, in both simulation and zero-shot real-world deployment, demonstrating strong constraint satisfaction and robust performance with minimal reward terms and domain randomization. The results indicate substantial engineering advantages: constraints generalize across morphologies, reduce tuning complexity, and enable programmable behavior changes (e.g., gait or obstacle negotiation) without extensive reward design. Overall, the framework offers a scalable, generalizable pathway to high-performance legged locomotion with reduced engineering effort and reliable sim-to-real transfer.

Abstract

Several earlier studies have shown impressive control performance in complex robotic systems by designing the controller using a neural network and training it with model-free reinforcement learning. However, these outstanding controllers with natural motion style and high task performance are developed through extensive reward engineering, which is a highly laborious and time-consuming process of designing numerous reward terms and determining suitable reward coefficients. In this work, we propose a novel reinforcement learning framework for training neural network controllers for complex robotic systems consisting of both rewards and constraints. To let the engineers appropriately reflect their intent to constraints and handle them with minimal computation overhead, two constraint types and an efficient policy optimization algorithm are suggested. The learning framework is applied to train locomotion controllers for several legged robots with different morphology and physical attributes to traverse challenging terrains. Extensive simulation and real-world experiments demonstrate that performant controllers can be trained with significantly less reward engineering, by tuning only a single reward coefficient. Furthermore, a more straightforward and intuitive engineering process can be utilized, thanks to the interpretability and generalizability of constraints. The summary video is available at https://youtu.be/KAlm3yskhvM.
Paper Structure (34 sections, 12 equations, 9 figures, 8 tables, 1 algorithm)

This paper contains 34 sections, 12 equations, 9 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Outline
  4. Related Work
  5. Control of complex robotic systems
  6. Legged robot locomotion
  7. Constrained reinforcement learning
  8. Background
  9. Constrained Markov Decision Process
  10. Learning Framework
  11. Constraint Types
  12. Probabilistic Constraints
  13. Average Constraints
  14. Policy optimization
  15. Adaptive constraint thresholding
  16. ...and 19 more sections

Figures (9)

  • Figure 1: Several legged robots trained with our learning framework to locomote in challenging terrains. The robots include six quadrupedal robots and one bipedal robot.
  • Figure 2: Overall framework that leverages both rewards and constraints to train locomotion controllers for various legged robots
  • Figure 3: Evaluation and analysis results of the proposed framework. (A) shows the total reward (left) and two costs (middle, right) of multiple robots. (B) shows each reward component's value to check the objective sensitivity property. Every reward has been standardized so that it ranges from -1 to 0. (C) shows the results to verify the constraint satisfaction property. Specifically, foot clearance (C-1), gait pattern (i.e., trot, pace, fast trot) (C-2), and joint angle (C-3) constraints were modified and tested in the simulation. The resulting foot clearances, foot contact patterns, and joint angle trajectories are plotted.
  • Figure 4: Real-world experiments with two quadruped robots: Raibo and Mini-cheetah. The robustness of the controllers was extensively tested in various harsh terrains both indoor and outdoor. Outdoor experiments were conducted on two long paths: a hiking route and a campus route. The robot's speed is calculated using the video's recorded time stamp and the measured length of the traversal path. The slope of the environment is measured using a digital angle gauge.
  • Figure 5: Real-world experiments of generating multiple motions by varying constraints. A robot walking with a trotting gait (top) was changed to locomote with a bounding gait (middle) or crawl beneath an overhanging obstacle (bottom) by modifying the gait pattern constraint or COM height with joint angle constraint.
  • ...and 4 more figures