Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MetaWorld: Skill Transfer and Composition in a Hierarchical World Model for Grounding High-Level Instructions

Yutong Shen, Hangxu Liu, Kailin Pei, Ruizhe Xia, Tongtong Feng

TL;DR

MetaWorld tackles the semantic-physical gap in humanoid loco-manipulation by introducing a hierarchical world model that decouples semantic planning from physical control. A Vision-Language Model provides task-conditioned expert weights, while a latent dynamics model and MPC-based execution realize actions, with a state-aware fusion mechanism to adapt in real time. The approach yields large performance gains on HumanoidBench, demonstrating improved task success and motion coherence through semantic parsing and dynamic expert composition. This modular framework enhances sample efficiency and robustness, enabling scalable grounding of high-level instructions into physically feasible skills.

Abstract

Humanoid robot loco-manipulation remains constrained by the semantic-physical gap. Current methods face three limitations: Low sample efficiency in reinforcement learning, poor generalization in imitation learning, and physical inconsistency in VLMs. We propose MetaWorld, a hierarchical world model that integrates semantic planning and physical control via expert policy transfer. The framework decouples tasks into a VLM-driven semantic layer and a latent dynamics model operating in a compact state space. Our dynamic expert selection and motion prior fusion mechanism leverages a pre-trained multi-expert policy library as transferable knowledge, enabling efficient online adaptation via a two-stage framework. VLMs serve as semantic interfaces, mapping instructions to executable skills and bypassing symbol grounding. Experiments on Humanoid-Bench show MetaWorld outperforms world model-based RL in task completion and motion coherence. Our code will be found at https://anonymous.4open.science/r/metaworld-2BF4/

MetaWorld: Skill Transfer and Composition in a Hierarchical World Model for Grounding High-Level Instructions

TL;DR

MetaWorld tackles the semantic-physical gap in humanoid loco-manipulation by introducing a hierarchical world model that decouples semantic planning from physical control. A Vision-Language Model provides task-conditioned expert weights, while a latent dynamics model and MPC-based execution realize actions, with a state-aware fusion mechanism to adapt in real time. The approach yields large performance gains on HumanoidBench, demonstrating improved task success and motion coherence through semantic parsing and dynamic expert composition. This modular framework enhances sample efficiency and robustness, enabling scalable grounding of high-level instructions into physically feasible skills.

Abstract

Humanoid robot loco-manipulation remains constrained by the semantic-physical gap. Current methods face three limitations: Low sample efficiency in reinforcement learning, poor generalization in imitation learning, and physical inconsistency in VLMs. We propose MetaWorld, a hierarchical world model that integrates semantic planning and physical control via expert policy transfer. The framework decouples tasks into a VLM-driven semantic layer and a latent dynamics model operating in a compact state space. Our dynamic expert selection and motion prior fusion mechanism leverages a pre-trained multi-expert policy library as transferable knowledge, enabling efficient online adaptation via a two-stage framework. VLMs serve as semantic interfaces, mapping instructions to executable skills and bypassing symbol grounding. Experiments on Humanoid-Bench show MetaWorld outperforms world model-based RL in task completion and motion coherence. Our code will be found at https://anonymous.4open.science/r/metaworld-2BF4/
Paper Structure (16 sections, 4 equations, 4 figures, 1 table)

This paper contains 16 sections, 4 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. World Models in Robotics
  4. Skill Transfer Learning
  5. VLM Applications in Real-time Task Parsing
  6. Method
  7. MetaWorld Hierarchical Architecture
  8. Semantic Planning and Symbol Grounding
  9. Dynamic Adaptation Mechanism
  10. Physical Execution and Online Optimization
  11. Theoretical Analysis and Implementation
  12. Experiment
  13. Experimental setup
  14. Validation of Hierarchical Architecture Effectiveness
  15. Ablation study and stability analysis
  16. ...and 1 more sections

Figures (4)

  • Figure 1: This figure demonstrates the effectiveness of our hierarchical world model architecture on both the walking expert policy and the door-opening task.
  • Figure 2: This illustrates the three-tier architecture of the MetaWorld framework: the semantic layer parses observation into executable skill sequences via a vision-language model; the skill transfer layer integrates expert policy priors through a hierarchical world model and enables dynamic adaptation; the physical layer performs precise control in a compact state space using a latent dynamics model.
  • Figure 3: We evaluate our method on locomotion (run, walk, stand) and manipulation (door) tasks against baselines TD-MPC2 and DreamerV3.
  • Figure 4: Ablation study results on the Door task (maximum reward and convergence steps).