World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning

TL;DR

This work tackles the limitation that LVLM-based embodied task planning lacks a robust internal world model. It introduces Dual Preference Optimization (D$^2$PO), jointly optimizing state prediction and action selection through preference learning to develop a predictive world model that aids planning. A tree-search data collection strategy enables automatic generation of trajectories and stepwise preference data without human annotations, facilitating scalable training. Empirical results on VoTa-Bench show that D$^2$PO, even at 7B parameters, outperforms strong baselines including GPT-4o in both success rate and planning efficiency, and generalizes well to unseen scenes, highlighting the practical impact of integrating world modeling into planning for embodied AI.

Abstract

Recent advances in large vision-language models (LVLMs) have shown promise for embodied task planning, yet they struggle with fundamental challenges like dependency constraints and efficiency. Existing approaches either solely optimize action selection or leverage world models during inference, overlooking the benefits of learning to model the world as a way to enhance planning capabilities. We propose Dual Preference Optimization (D$^2$PO), a new learning framework that jointly optimizes state prediction and action selection through preference learning, enabling LVLMs to understand environment dynamics for better planning. To automatically collect trajectories and stepwise preference data without human annotation, we introduce a tree search mechanism for extensive exploration via trial-and-error. Extensive experiments on VoTa-Bench demonstrate that our D$^2$PO-based method significantly outperforms existing methods and GPT-4o when applied to Qwen2-VL (7B), LLaVA-1.6 (7B), and LLaMA-3.2 (11B), achieving superior task success rates with more efficient execution paths.

Figures (10)

• Figure 1: Overview of D$^2$PO: World modeling enables better embodied task planning through joint preference optimization of state prediction and action selection.
• Figure 2: Our method consists of two dimensions: (a) Data Exploration via Step-wise Tree Search (Sec \ref{['sec:data']}), which collects preference data through sampling and selecting potential actions, iterative tree expansion, and trajectory backtracking; (b) Dual Preference Optimization (D$^2$PO) framework (Sec \ref{['sec:d2po']}) that leverages the collected preference pairs to jointly optimize action selection and state prediction.
• Figure 3: Analysis of data scale and model scale.
• Figure 4: Success rates (SR) of action-conditioned and goal-directed world models across seen and unseen scenarios.
• Figure 5: Comparison of ALFRED, LoTa-Bench, and VoTa-Bench in the task "Place a cold tomato in the sink". (a) ALFRED emphasizes high-level task planning with human-written step-by-step instructions, breaking the task into subgoals like "Cool Tomato" (step 4). (b) LoTa-Bench provides only goal instructions and decomposes tasks into fine-grained low-level actions (e.g., "Open Fridge", "PutDown Tomato", etc.; steps 4–10) but lacks guidance from visual input, relying on predefined executable actions, choosing actions based on maximum logits to ensure they are valid in the simulation. (c) VoTa-Bench extends LoTa-Bench by incorporating egocentric visual observations, requiring models to generate open-domain actions based on visual information to handle both seen and unseen environments.