Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evolutionary Continuous Adaptive RL-Powered Co-Design for Humanoid Chin-Up Performance

Tianyi Jin, Melya Boukheddimi, Rohit Kumar, Gabriele Fadini, Frank Kirchner

TL;DR

The paper tackles the problem of co-designing both the morphology and control policy of humanoid robots to perform high-demand tasks such as chin-ups. It introduces EA-CoRL, a bi-level framework combining CMA-ES for design search with continuous policy adaptation via PPO, enabling a policy that generalizes across evolving designs. Experimental validation on the RH5 humanoid shows that continuous policy refinement yields higher rewards and broader design exploration than a baseline pre-training + fine-tuning approach, and identifies gear-ratio configurations that unlock previously infeasible motions without added mass. The results demonstrate the value of integrating design optimization with ongoing policy adaptation, supporting broader applicability to other humanoid and legged systems and highlighting practical implications for hardware-efficient, robust co-design in dynamic tasks.

Abstract

Humanoid robots have seen significant advancements in both design and control, with a growing emphasis on integrating these aspects to enhance overall performance. Traditionally, robot design has followed a sequential process, where control algorithms are developed after the hardware is finalized. However, this can be myopic and prevent robots to fully exploit their hardware capabilities. Recent approaches advocate for co-design, optimizing both design and control in parallel to maximize robotic capabilities. This paper presents the Evolutionary Continuous Adaptive RL-based Co-Design (EA-CoRL) framework, which combines reinforcement learning (RL) with evolutionary strategies to enable continuous adaptation of the control policy to the hardware. EA-CoRL comprises two key components: Design Evolution, which explores the hardware choices using an evolutionary algorithm to identify efficient configurations, and Policy Continuous Adaptation, which fine-tunes a task-specific control policy across evolving designs to maximize performance rewards. We evaluate EA-CoRL by co-designing the actuators (gear ratios) and control policy of the RH5 humanoid for a highly dynamic chin-up task, previously unfeasible due to actuator limitations. Comparative results against state-of-the-art RL-based co-design methods show that EA-CoRL achieves higher fitness score and broader design space exploration, highlighting the critical role of continuous policy adaptation in robot co-design.

Evolutionary Continuous Adaptive RL-Powered Co-Design for Humanoid Chin-Up Performance

TL;DR

The paper tackles the problem of co-designing both the morphology and control policy of humanoid robots to perform high-demand tasks such as chin-ups. It introduces EA-CoRL, a bi-level framework combining CMA-ES for design search with continuous policy adaptation via PPO, enabling a policy that generalizes across evolving designs. Experimental validation on the RH5 humanoid shows that continuous policy refinement yields higher rewards and broader design exploration than a baseline pre-training + fine-tuning approach, and identifies gear-ratio configurations that unlock previously infeasible motions without added mass. The results demonstrate the value of integrating design optimization with ongoing policy adaptation, supporting broader applicability to other humanoid and legged systems and highlighting practical implications for hardware-efficient, robust co-design in dynamic tasks.

Abstract

Humanoid robots have seen significant advancements in both design and control, with a growing emphasis on integrating these aspects to enhance overall performance. Traditionally, robot design has followed a sequential process, where control algorithms are developed after the hardware is finalized. However, this can be myopic and prevent robots to fully exploit their hardware capabilities. Recent approaches advocate for co-design, optimizing both design and control in parallel to maximize robotic capabilities. This paper presents the Evolutionary Continuous Adaptive RL-based Co-Design (EA-CoRL) framework, which combines reinforcement learning (RL) with evolutionary strategies to enable continuous adaptation of the control policy to the hardware. EA-CoRL comprises two key components: Design Evolution, which explores the hardware choices using an evolutionary algorithm to identify efficient configurations, and Policy Continuous Adaptation, which fine-tunes a task-specific control policy across evolving designs to maximize performance rewards. We evaluate EA-CoRL by co-designing the actuators (gear ratios) and control policy of the RH5 humanoid for a highly dynamic chin-up task, previously unfeasible due to actuator limitations. Comparative results against state-of-the-art RL-based co-design methods show that EA-CoRL achieves higher fitness score and broader design space exploration, highlighting the critical role of continuous policy adaptation in robot co-design.

Paper Structure

This paper contains 25 sections, 8 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Advances in Co-Design Approaches
  3. Reinforcement Learning-Based Co-Design
  4. Methodology
  5. Design Evolution: Outer Loop (Red Block)
  6. Continuous Adaptation: Inner Loop (Blue Block)
  7. Base Policy $\pi_{\theta_0}$
  8. Fine-tuned Policy $\pi_{\theta^*}$
  9. Task-Adapted Policy $\pi_{\theta}$
  10. RL Training
  11. Best Robot Design $d^*$:
  12. EXPERIMENTAL SETUP
  13. RH5 Whole-Body Humanoid
  14. Chin-Up: Evaluating High-Force, Dynamic Performance
  15. EA-CoRL Experimental Details for RH5's Chin-Up Task
  16. ...and 10 more sections

Figures (6)

  • Figure 1: Chin-up RL policies on the RH5 humanoid designs obtained through EA-CoRL.
  • Figure 2: Overview of the EA-CoRL framework methodology.
  • Figure 3: Original and obtained best RH5 gear ratio factors for the chin-up task using EA-CoRL (ours) and PT-FT methods.
  • Figure 4: Heat-maps of RH5 gear-ratio factor's correlations for chin-up task. Blue stars represent the best design using the EA-CoRL method and green stars represent the best design using the PT-FT method.
  • Figure 5: Comparison of EA-CoRL and PT-FT methods based on mean $\pm$ standard deviation of fitness scores across the entire Design Evolution process, averaged over six random seeds. The x-axis represents the cumulative number of design's evolution.
  • ...and 1 more figures