DreamWorld: Unified World Modeling in Video Generation

TL;DR

A unified framework that integrates complementary world knowledge into video generators via a Joint World Modeling Paradigm, and a proposal to progressively regulate world-level constraints during training and enforce learned world priors at inference is proposed.

Abstract

Despite impressive progress in video generation, existing models remain limited to surface-level plausibility, lacking a coherent and unified understanding of the world. Prior approaches typically incorporate only a single form of world-related knowledge or rely on rigid alignment strategies to introduce additional knowledge. However, aligning the single world knowledge is insufficient to constitute a world model that requires jointly modeling multiple heterogeneous dimensions (e.g., physical commonsense, 3D and temporal consistency). To address this limitation, we introduce \textbf{DreamWorld}, a unified framework that integrates complementary world knowledge into video generators via a \textbf{Joint World Modeling Paradigm}, jointly predicting video pixels and features from foundation models to capture temporal dynamics, spatial geometry, and semantic consistency. However, naively optimizing these heterogeneous objectives can lead to visual instability and temporal flickering. To mitigate this issue, we propose \textit{Consistent Constraint Annealing (CCA)} to progressively regulate world-level constraints during training, and \textit{Multi-Source Inner-Guidance} to enforce learned world priors at inference. Extensive evaluations show that DreamWorld improves world consistency, outperforming Wan2.1 by 2.26 points on VBench. Code will be made publicly available at \href{https://github.com/ABU121111/DreamWorld}{\textcolor{mypink}{\textbf{Github}}}.

Figures (6)

• Figure 1: Limitations of extending VideoREPA to multi-source knowledge, leading to structural implausibility and unnatural distortion. The physics score(PC) on the Videophy benchmark dropped from 29.7 to 24.1.
• Figure 2: Overview of DreamWorld.(a) Training: Expert Models are first employed to extract multimodal features, which are then noise-added and concatenated before being fused through a linear layer $\mathbf{W_{in}^+}$. The resulting prediction is mapped via another linear layer $\mathbf{W_{out}^+}$ to jointly predict appearance and world knowledge, with a Dream Loss constrained by Consistency Constraint Annealing (CCA) to ensure generation fidelity. (b) Inference: We introduce Multi-Source Inner-Guidance, a mechanism that leverages inherent noise features to direct the final video generation process.
• Figure 3: Qualitative comparison of world consistency. DreamWorld outperforms baselines by maintaining semantic realism, spatial integrity, and temporal precision. In contrast, competitor models frequently exhibit geometric penetrations and uncanny distortions
• Figure 4: Influence of loss weights $\lambda$. Quantitative comparison of generation quality and semantic alignment across different weight settings. The results indicate that $\lambda=0.2$ yields the best trade-off.
• Figure 5: Effectiveness of CCA. Without CCA, the generated videos suffer from severe flickering and abnormal highlighting artifacts, leading to degraded visual quality.