Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control

Weidong Huang, Zhehan Li, Hangxin Liu, Biao Hou, Yao Su, Jingwen Zhang

TL;DR

This paper finds that off-policy Soft Actor-Critic (SAC), with large-batch update and a high Update-To-Data (UTD) ratio, reliably supports large-scale pretraining of humanoid locomotion policies, achieving zero-shot deployment on real robots.

Abstract

Reinforcement learning (RL) is widely used for humanoid control, with on-policy methods such as Proximal Policy Optimization (PPO) enabling robust training via large-scale parallel simulation and, in some cases, zero-shot deployment to real robots. However, the low sample efficiency of on-policy algorithms limits safe adaptation to new environments. Although off-policy RL and model-based RL have shown improved sample efficiency, the gap between large-scale pretraining and efficient finetuning on humanoids still exists. In this paper, we find that off-policy Soft Actor-Critic (SAC), with large-batch update and a high Update-To-Data (UTD) ratio, reliably supports large-scale pretraining of humanoid locomotion policies, achieving zero-shot deployment on real robots. For adaptation, we demonstrate that these SAC-pretrained policies can be finetuned in new environments and out-of-distribution tasks using model-based methods. Data collection in the new environment executes a deterministic policy while stochastic exploration is instead confined to a physics-informed world model. This separation mitigates the risks of random exploration during adaptation while preserving exploratory coverage for improvement. Overall, the approach couples the wall-clock efficiency of large-scale simulation during pretraining with the sample efficiency of model-based learning during fine-tuning.

Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control

TL;DR

This paper finds that off-policy Soft Actor-Critic (SAC), with large-batch update and a high Update-To-Data (UTD) ratio, reliably supports large-scale pretraining of humanoid locomotion policies, achieving zero-shot deployment on real robots.

Abstract

Reinforcement learning (RL) is widely used for humanoid control, with on-policy methods such as Proximal Policy Optimization (PPO) enabling robust training via large-scale parallel simulation and, in some cases, zero-shot deployment to real robots. However, the low sample efficiency of on-policy algorithms limits safe adaptation to new environments. Although off-policy RL and model-based RL have shown improved sample efficiency, the gap between large-scale pretraining and efficient finetuning on humanoids still exists. In this paper, we find that off-policy Soft Actor-Critic (SAC), with large-batch update and a high Update-To-Data (UTD) ratio, reliably supports large-scale pretraining of humanoid locomotion policies, achieving zero-shot deployment on real robots. For adaptation, we demonstrate that these SAC-pretrained policies can be finetuned in new environments and out-of-distribution tasks using model-based methods. Data collection in the new environment executes a deterministic policy while stochastic exploration is instead confined to a physics-informed world model. This separation mitigates the risks of random exploration during adaptation while preserving exploratory coverage for improvement. Overall, the approach couples the wall-clock efficiency of large-scale simulation during pretraining with the sample efficiency of model-based learning during fine-tuning.
Paper Structure (49 sections, 21 equations, 16 figures, 7 tables, 1 algorithm)

This paper contains 49 sections, 21 equations, 16 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Model-based RL Problem.
  5. Physics-Informed World Model for Robots
  6. Large-scale Pretraining and Efficient Fine-tuning
  7. Large-scale Policy Pretraining
  8. Policy Optimization.
  9. Large-Batch Updates and High UTD.
  10. World Model Pretraining
  11. Offline Pretraining from Logged Transitions
  12. Policy and World Model Finetuning
  13. Deployment & Data Collection.
  14. World-Model Fine-Tuning.
  15. Exploration in Physics-informed World-Model.
  16. ...and 34 more sections

Figures (16)

  • Figure 1: Large-scale pretraIning and efficient FineTuning (LIFT) Framework. In stage (i), we implement SAC in JAX to support large-batch update and high utd, achieving fast, robust convergence in massively parallel simulation and zero-shot deployment to a real humanoid in outdoor experiments. In stage (ii), we pretrain a physics-informed world model on the SAC data, combining Lagrangian dynamics with a residual predictor to capture contact forces and other unmodeled effects. In stage (iii), we finetune both the policy and the world model to new environments while executing only deterministic actions in the environment. Stochastic exploration is confined to rollouts within the world model. This framework enhances both the safety and efficiency of finetuning.
  • Figure 2: Results of finetuning Booster T1 robot with varying target speeds. The black dashed line represents the target velocity for each task. Results are averaged over 8 random seeds.
  • Figure 3: Real-world finetuning progression on the Booster T1 humanoid. A video demonstration is available on our project website.
  • Figure 4: Ablation of the pretraining on Booster T1 (target forward speed $=1.5\,$m/s). Results are averaged over 8 random seeds.
  • Figure 5: Ablation of Physics informed World Model on Booster T1 (target speed $=1.5\,$m/s). Results are averaged over 8 random seeds.
  • ...and 11 more figures