Bridging the Gap Between Multimodal Foundation Models and World Models

TL;DR

This work investigates how to bridge multimodal foundation models (MFMs) with world models by pursuing discriminative and generative capabilities. It introduces a two-part program: Part I enhances MFMs with structured reasoning (causality, counterfactuals, spatiotemporal reasoning) for perception tasks, and Part II develops generative, controllable, and interactive 4D content across images, videos, and 4D scenes. Key contributions include efficient adaptation techniques (subspace training and Kronecker adaptation), counterfactual prompting for robust multimodal representations, compositional reasoning via causality, graph-informed multimodal generation, diffusion-based discriminative methods, graph-augmented transformers for structured reasoning, and a suite of benchmarks (MMWorld, VLM4D, Morpho4D) to evaluate perception, reasoning, and generation. The work demonstrates state-of-the-art or strong results across several tasks, including few-shot adaptation, compositional matching, spatiotemporal awareness, and interactive 4D scene construction, highlighting practical pathways toward scalable, controllable, and cognitively capable multimodal world models.

Abstract

Humans understand the world through the integration of multiple sensory modalities, enabling them to perceive, reason about, and imagine dynamic physical processes. Inspired by this capability, multimodal foundation models (MFMs) have emerged as powerful tools for multimodal understanding and generation. However, today's MFMs fall short of serving as effective world models. They lack the essential ability such as perform counterfactual reasoning, simulate dynamics, understand the spatiotemporal information, control generated visual outcomes, and perform multifaceted reasoning. We investigates what it takes to bridge the gap between multimodal foundation models and world models. We begin by improving the reasoning capabilities of MFMs through discriminative tasks and equipping MFMs with structured reasoning skills, such as causal inference, counterfactual thinking, and spatiotemporal reasoning, enabling them to go beyond surface correlations and understand deeper relationships within visual and textual data. Next, we explore generative capabilities of multimodal foundation models across both image and video modalities, introducing new frameworks for structured and controllable generation. Our approaches incorporate scene graphs, multimodal conditioning, and multimodal alignment strategies to guide the generation process, ensuring consistency with high-level semantics and fine-grained user intent. We further extend these techniques to controllable 4D generation, enabling interactive, editable, and morphable object synthesis over time and space.

Figures (43)

• Figure 1: The results are measured using the vision transformer (ViT-B-224/32) via CLIP pretraining across the average of 20 image classification datasets. Our method places in the topleft corner and achieves the best tradeoff between accuracy and parameter efficiency. The color of points and numbers denote the performance-efficiency (PE) metric (higher is better).
• Figure 2: $\boldsymbol{A_i}$ denotes the shared weight matrix, with $i \in\{1, \ldots, n\}$. $\boldsymbol{B_i}$ is decomposed into two low-rank matrices $\boldsymbol{u}_{\boldsymbol{i}}$ and $\boldsymbol{v}_{\boldsymbol{i}}$. $\boldsymbol{h}$ is the output of the selected ViT submodule. $\boldsymbol{x}$ is the input to the submodule. During model adaptation process, only matrices $\boldsymbol{A_i}$, $\boldsymbol{u}_{\boldsymbol{i}}$, and $\boldsymbol{v}_{\boldsymbol{i}}$ receive gradients to improve parameter efficiency.
• Figure 3: Validation Accuracy vs. Subspace Dimension $d$ of MLP and the attention module for Supervised ViT on CIFAR100. The local intrinsic dimension $d_t$ of the attention module is lower than that of the MLP.
• Figure 4: A conceptual overview of counterfactual prompt learning. CPL constructs counterfactuals by identifying non-spurious feature change that causally causes the prompt change. In this case, the "barn" feature is the essential cause between Prompt A and B.
• Figure 5: The counterfactual prompt learning framework. We freeze the vision encoder $F$ and the text encoder $G$, and only optimize the task-agnostic prompts and the instance-conditioned net $M$ (blue blocks). Please refer to Section \ref{['sec:overview']} for the explanation.