ReWorld: Multi-Dimensional Reward Modeling for Embodied World Models

TL;DR

ReWorld tackles the Physics Uncanny Valley in video-based embodied world models by introducing a multi-dimensional reward framework (HERO) and a tractable flow-based policy optimization (HERO-FPO). It collects a large 4D Embodied Preference Dataset and trains a four-headed reward model that decouples physical, embodiment, task, and visual signals, enabling precise alignment with human preferences. The CFM-Likelihood proxy makes PPO-style updates feasible for high-resolution flow models, allowing RLHF to refine embodied rollouts. The authors also propose ReWorldBench to rigorously evaluate physical realism, embodiment, task execution, and visual fidelity, demonstrating significant improvements over baselines and strong human preference endorsements. Together, these contributions advance practical, physically grounded, and visually convincing embodied world models suitable for downstream robotic tasks.

Abstract

Recently, video-based world models that learn to simulate the dynamics have gained increasing attention in robot learning. However, current approaches primarily emphasize visual generative quality while overlooking physical fidelity, dynamic consistency, and task logic, especially for contact-rich manipulation tasks, which limits their applicability to downstream tasks. To this end, we introduce ReWorld, a framework aimed to employ reinforcement learning to align the video-based embodied world models with physical realism, task completion capability, embodiment plausibility and visual quality. Specifically, we first construct a large-scale (~235K) video preference dataset and employ it to train a hierarchical reward model designed to capture multi-dimensional reward consistent with human preferences. We further propose a practical alignment algorithm that post-trains flow-based world models using this reward through a computationally efficient PPO-style algorithm. Comprehensive experiments and theoretical analysis demonstrate that ReWorld significantly improves the physical fidelity, logical coherence, embodiment and visual quality of generated rollouts, outperforming previous methods.

Figures (3)

• Figure 1: Our proposed ReWorld bridges the gap of video-based embodied world models with physical realism, task completion capability, embodiment plausibility, and visual quality. Each of the four dimensions shown is rated on a scale of 1 (poor) to 6 (excellent), where higher scores indicate better performance.
• Figure 2: Overview of the ReWorld framework. (a) We employ a VLM-driven annotation system to generate the 4-dimensional embodied preference dataset. (b) Building upon this dataset, we train the multi-dimensional reward model HERO based on the hierarchical feature space of InternVideo2 wang2024internvideo2. (c) We detail the reinforcement learning pipeline HERO-FPO to refine the generative policy with the learned multi-dimensional reward signal. (d) We introduce ReWorldBench as a specialized benchmark to evaluate embodied world models.
• Figure 3: Qualitative comparisons on ReWorldBench. Our proposed ReWorld model achieves the best generative results for all the multi-dimensional metrics compared with the baseline video generation models.