Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test

TL;DR

WoW-World-Eval introduces a robotics-centric, multi-faceted benchmark to evaluate embodied world models across perception, prediction, planning, execution, and generalization, using 609 robot-manipulation samples and 22 metrics. It couples a Human Turing Test with an IDM-based machine Turing Test to assess perceptual fidelity and physical executability, revealing a gap between high visual realism and real-world action. Experiments across diverse models show perceptual metrics can align with human preferences, yet long-horizon planning and real-world execution remain major bottlenecks, with IDM success remaining low for most systems. The framework provides a principled, scalable standard for advancing physically grounded, generalizable embodied world models in robotics.

Abstract

As world models gain momentum in Embodied AI, an increasing number of works explore using video foundation models as predictive world models for downstream embodied tasks like 3D prediction or interactive generation. However, before exploring these downstream tasks, video foundation models still have two critical questions unanswered: (1) whether their generative generalization is sufficient to maintain perceptual fidelity in the eyes of human observers, and (2) whether they are robust enough to serve as a universal prior for real-world embodied agents. To provide a standardized framework for answering these questions, we introduce the Embodied Turing Test benchmark: WoW-World-Eval (Wow,wo,val). Building upon 609 robot manipulation data, Wow-wo-val examines five core abilities, including perception, planning, prediction, generalization, and execution. We propose a comprehensive evaluation protocol with 22 metrics to assess the models' generation ability, which achieves a high Pearson Correlation between the overall score and human preference (>0.93) and establishes a reliable foundation for the Human Turing Test. On Wow-wo-val, models achieve only 17.27 on long-horizon planning and at best 68.02 on physical consistency, indicating limited spatiotemporal consistency and physical reasoning. For the Inverse Dynamic Model Turing Test, we first use an IDM to evaluate the video foundation models' execution accuracy in the real world. However, most models collapse to $\approx$ 0% success, while WoW maintains a 40.74% success rate. These findings point to a noticeable gap between the generated videos and the real world, highlighting the urgency and necessity of benchmarking World Model in Embodied AI.

Figures (10)

• Figure 1: The Overview of WoW-World-Eval.(Top-left) A multi-faceted Metrics suite evaluates generated videos across five dimensions: Video Quality, Instruction Understanding, Planning Reasoning, Physical Law, and Execution Accuracy. (Top-center) These metrics align with five core Abilities of embodied world models: Perception, Planning, Prediction, Execution, and Generalization. (Top-right) Performance gaps across state-of-the-art models. (Bottom) The benchmark follows the embodied world model pipeline from Perception to Generalization.
• Figure 2: Overview of WoW-World-Eval Data Construction Pipeline and Data Statistics.(Left) Public and in-house data are cleaned sequentially by GPT and human annotators to produce high-quality samples consisting of an initial image, prompt, ground-truth video, and annotated keypoints. (Right) Data distribution across five different dimensions from overall to fine-grained.
• Figure 3: Side-by-side comparison: (a) qualitative results by different models; (b) display a metric–preference correlation.
• Figure 4: Trending between WoW-World-Eval Overall Score and Deceive Human Ratio.
• Figure 5: Overview of WoW-World-Eval Metrics.