Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model

Wenyuan Chen, Jinbang Huang, Oscar Pang, Zhiyuan Li, Xiao Hu, Lingfeng Zhang, Zhanguang Zhang, Mark Coates, Tongtong Cao, Xingyue Quan, Yingxue Zhang

TL;DR

This work introduces a Hierarchical World Model (H-WM) that jointly predicts symbolic state transitions and latent visual subgoals to guide long-horizon Vision–Language–Action (VLA) robotic tasks. It combines a fine-tuned LLM-based logical world model with a latent-feature visual world model, enabling stable, perceptually grounded planning and execution through subtask completion signals and structured attention. The authors build the LIBERO-Logic dataset, train on LIBERO-LoHo benchmarks, and demonstrate that bilevel guidance significantly improves Q-Score and Success Rate over end-to-end, language-guided, and logic-only baselines, with ablations highlighting the value of visual grounding. This approach offers a scalable way to bridge symbolic reasoning and perceptual grounding, improving robustness for complex manipulation tasks across extended horizons.

Abstract

World models are becoming central to robotic planning and control, as they enable prediction of future state transitions. Existing approaches often emphasize video generation or natural language prediction, which are difficult to directly ground in robot actions and suffer from compounding errors over long horizons. Traditional task and motion planning relies on symbolic logic world models, such as planning domains, that are robot-executable and robust for long-horizon reasoning. However, these methods typically operate independently of visual perception, preventing synchronized symbolic and perceptual state prediction. We propose a Hierarchical World Model (H-WM) that jointly predicts logical and visual state transitions within a unified bilevel framework. H-WM combines a high-level logical world model with a low-level visual world model, integrating the robot-executable, long-horizon robustness of symbolic reasoning with perceptual grounding from visual observations. The hierarchical outputs provide stable and consistent intermediate guidance for long-horizon tasks, mitigating error accumulation and enabling robust execution across extended task sequences. To train H-WM, we introduce a robotic dataset that aligns robot motion with symbolic states, actions, and visual observations. Experiments across vision-language-action (VLA) control policies demonstrate the effectiveness and generality of the approach.

H-WM: Robotic Task and Motion Planning Guided by Hierarchical World Model

TL;DR

This work introduces a Hierarchical World Model (H-WM) that jointly predicts symbolic state transitions and latent visual subgoals to guide long-horizon Vision–Language–Action (VLA) robotic tasks. It combines a fine-tuned LLM-based logical world model with a latent-feature visual world model, enabling stable, perceptually grounded planning and execution through subtask completion signals and structured attention. The authors build the LIBERO-Logic dataset, train on LIBERO-LoHo benchmarks, and demonstrate that bilevel guidance significantly improves Q-Score and Success Rate over end-to-end, language-guided, and logic-only baselines, with ablations highlighting the value of visual grounding. This approach offers a scalable way to bridge symbolic reasoning and perceptual grounding, improving robustness for complex manipulation tasks across extended horizons.

Abstract

World models are becoming central to robotic planning and control, as they enable prediction of future state transitions. Existing approaches often emphasize video generation or natural language prediction, which are difficult to directly ground in robot actions and suffer from compounding errors over long horizons. Traditional task and motion planning relies on symbolic logic world models, such as planning domains, that are robot-executable and robust for long-horizon reasoning. However, these methods typically operate independently of visual perception, preventing synchronized symbolic and perceptual state prediction. We propose a Hierarchical World Model (H-WM) that jointly predicts logical and visual state transitions within a unified bilevel framework. H-WM combines a high-level logical world model with a low-level visual world model, integrating the robot-executable, long-horizon robustness of symbolic reasoning with perceptual grounding from visual observations. The hierarchical outputs provide stable and consistent intermediate guidance for long-horizon tasks, mitigating error accumulation and enabling robust execution across extended task sequences. To train H-WM, we introduce a robotic dataset that aligns robot motion with symbolic states, actions, and visual observations. Experiments across vision-language-action (VLA) control policies demonstrate the effectiveness and generality of the approach.
Paper Structure (26 sections, 4 equations, 3 figures, 2 tables)

This paper contains 26 sections, 4 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. World Model
  4. Vision–Language–Action Models
  5. Task and Motion Planning
  6. Preliminary
  7. Symbolic Formalization
  8. Visual Representation
  9. Methodology
  10. Logic World Model
  11. Visual World Model
  12. Pipeline for VLA under Hierarchical World Model Guidance
  13. Subtask Completion and Transition Prediction
  14. Experiments
  15. Training Dataset Generation
  16. ...and 11 more sections

Figures (3)

  • Figure 1: The proposed hierarchical world model jointly captures state transitions in both symbolic and perceptual spaces to guide robot motion policies. The logical world model performs long-horizon reasoning to predict logical states and action sequences. Conditioned on the previous observation, executed action, and predicted next logical state, the visual world model generates perceptually grounded sub-goal images that translate symbolic decisions into visual sub-goals. The low-level policy consumes the logical action, logical state, visual sub-goal, or their combination to produce continuous robot motions, enabling consistent task execution while maintaining physical feasibility.
  • Figure 2: (a) Overview of the proposed visual world model. The model comprises an understanding expert that processes the current scene and language prompt, and a prediction expert that generates goal-state encoded feature representations to guide low-level motion control. (b) Overview of the proposed sub-goal VLA. The understanding expert encodes multimodal perceptual features, while the goal expert processes the predicted goal-state representations. Integrating explicit goal-state information enables more accurate planning and execution in complex environments.
  • Figure 3: Case Study of vanilla $\pi_0.5$ and our H-WM-guided policy on long-horizon tasks.Case 1: Vanilla $\pi_0.5$ fails to reason over the full task horizon and prematurely closes the cabinet before placing the required target object, whereas our method successfully completes the task using bilevel guidance.Case 2: Vanilla $\pi_{0.5}$ selects an incorrect object, while our approach correctly identifies and manipulates the target, benefiting from future visual subgoal guidance that provides strong cues about target appearance and state transitions.Case 3: Vanilla $\pi_{0.5}$ omits critical intermediate steps due to incomplete task understanding; in contrast, logical state transitions combined with explicit visual grounding provide step-by-step guidance, enabling successful task execution.