Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SmallWorlds: Assessing Dynamics Understanding of World Models in Isolated Environments

Xinyi Li, Zaishuo Xia, Weyl Lu, Chenjie Hao, Yubei Chen

TL;DR

This work introduces the SmallWorld benchmark to rigorously evaluate world models in isolation, removing reward signals and using fully observable, controlled dynamics to reveal fundamental predictive and physical reasoning capabilities. By testing four architectures (RSSM, Transformer, Diffusion, Neural ODE) across six domains, it demonstrates clear differences in long-horizon stability and domain-specific strengths, notably showing diffusion-based and neural-ODE approaches excelling in structured physics while transformer-based models offer robust sequence modeling. The study also highlights trade-offs between robustness and precision and emphasizes the need for principled evaluation beyond policy-driven, reward-based metrics. Overall, SmallWorld provides a valuable, interpretable testbed to guide future advances in dynamics understanding and representation learning for world models.

Abstract

Current world models lack a unified and controlled setting for systematic evaluation, making it difficult to assess whether they truly capture the underlying rules that govern environment dynamics. In this work, we address this open challenge by introducing the SmallWorld Benchmark, a testbed designed to assess world model capability under isolated and precisely controlled dynamics without relying on handcrafted reward signals. Using this benchmark, we conduct comprehensive experiments in the fully observable state space on representative architectures including Recurrent State Space Model, Transformer, Diffusion model, and Neural ODE, examining their behavior across six distinct domains. The experimental results reveal how effectively these models capture environment structure and how their predictions deteriorate over extended rollouts, highlighting both the strengths and limitations of current modeling paradigms and offering insights into future improvement directions in representation learning and dynamics modeling.

SmallWorlds: Assessing Dynamics Understanding of World Models in Isolated Environments

TL;DR

This work introduces the SmallWorld benchmark to rigorously evaluate world models in isolation, removing reward signals and using fully observable, controlled dynamics to reveal fundamental predictive and physical reasoning capabilities. By testing four architectures (RSSM, Transformer, Diffusion, Neural ODE) across six domains, it demonstrates clear differences in long-horizon stability and domain-specific strengths, notably showing diffusion-based and neural-ODE approaches excelling in structured physics while transformer-based models offer robust sequence modeling. The study also highlights trade-offs between robustness and precision and emphasizes the need for principled evaluation beyond policy-driven, reward-based metrics. Overall, SmallWorld provides a valuable, interpretable testbed to guide future advances in dynamics understanding and representation learning for world models.

Abstract

Current world models lack a unified and controlled setting for systematic evaluation, making it difficult to assess whether they truly capture the underlying rules that govern environment dynamics. In this work, we address this open challenge by introducing the SmallWorld Benchmark, a testbed designed to assess world model capability under isolated and precisely controlled dynamics without relying on handcrafted reward signals. Using this benchmark, we conduct comprehensive experiments in the fully observable state space on representative architectures including Recurrent State Space Model, Transformer, Diffusion model, and Neural ODE, examining their behavior across six distinct domains. The experimental results reveal how effectively these models capture environment structure and how their predictions deteriorate over extended rollouts, highlighting both the strengths and limitations of current modeling paradigms and offering insights into future improvement directions in representation learning and dynamics modeling.

Paper Structure

This paper contains 13 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. The SmallWorld Benchmark
  4. Design Factors
  5. Physics Understanding
  6. Geometrical Understanding
  7. Combined with Other Benchmarks
  8. Experiments
  9. Baseline Architectures
  10. Multi-Domain Capability Evaluation
  11. Long-horizon Imagination
  12. Discussion and Conclusion
  13. Physical Understanding Task Setup

Figures (4)

  • Figure 1: Physical Understanding Tasks Visualization
  • Figure 2: Geometrical Tasks Visualization
  • Figure 3: Radar plot illustrating world model capability across task domains, represented as prediction error ratios relative to DreamerV3 and normalized to the maximum value. Left: performance across six task domains; Right: performance on physical reasoning tasks.
  • Figure 4: Prediction error as a function of imagination horizon across three representative tasks.