Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WorldGPT: Empowering LLM as Multimodal World Model

Zhiqi Ge, Hongzhe Huang, Mingze Zhou, Juncheng Li, Guoming Wang, Siliang Tang, Yueting Zhuang

TL;DR

WorldGPT introduces a generalist multimodal world model trained on millions of videos, integrating multimodal encoders, an LLM-based state-prediction module, and decoders to handle arbitrary modality transitions. A novel cognitive architecture with memory, knowledge retrieval, and a ContextReflector grounds predictions and supports long-horizon tasks, while WorldNet provides large-scale pretraining and curated evaluation data. The framework demonstrates strong state-transition modeling, and its use as a world simulator enables efficient dream tuning of multimodal agents with synthetic instructions that rival real-data fine-tuning in effectiveness, with significant gains in efficiency. Together, these components offer a scalable path toward universal multimodal world modeling and practical guidance for training and evaluating such systems.

Abstract

World models are progressively being employed across diverse fields, extending from basic environment simulation to complex scenario construction. However, existing models are mainly trained on domain-specific states and actions, and confined to single-modality state representations. In this paper, We introduce WorldGPT, a generalist world model built upon Multimodal Large Language Model (MLLM). WorldGPT acquires an understanding of world dynamics through analyzing millions of videos across various domains. To further enhance WorldGPT's capability in specialized scenarios and long-term tasks, we have integrated it with a novel cognitive architecture that combines memory offloading, knowledge retrieval, and context reflection. As for evaluation, we build WorldNet, a multimodal state transition prediction benchmark encompassing varied real-life scenarios. Conducting evaluations on WorldNet directly demonstrates WorldGPT's capability to accurately model state transition patterns, affirming its effectiveness in understanding and predicting the dynamics of complex scenarios. We further explore WorldGPT's emerging potential in serving as a world simulator, helping multimodal agents generalize to unfamiliar domains through efficiently synthesising multimodal instruction instances which are proved to be as reliable as authentic data for fine-tuning purposes. The project is available on \url{https://github.com/DCDmllm/WorldGPT}.

WorldGPT: Empowering LLM as Multimodal World Model

TL;DR

WorldGPT introduces a generalist multimodal world model trained on millions of videos, integrating multimodal encoders, an LLM-based state-prediction module, and decoders to handle arbitrary modality transitions. A novel cognitive architecture with memory, knowledge retrieval, and a ContextReflector grounds predictions and supports long-horizon tasks, while WorldNet provides large-scale pretraining and curated evaluation data. The framework demonstrates strong state-transition modeling, and its use as a world simulator enables efficient dream tuning of multimodal agents with synthetic instructions that rival real-data fine-tuning in effectiveness, with significant gains in efficiency. Together, these components offer a scalable path toward universal multimodal world modeling and practical guidance for training and evaluating such systems.

Abstract

World models are progressively being employed across diverse fields, extending from basic environment simulation to complex scenario construction. However, existing models are mainly trained on domain-specific states and actions, and confined to single-modality state representations. In this paper, We introduce WorldGPT, a generalist world model built upon Multimodal Large Language Model (MLLM). WorldGPT acquires an understanding of world dynamics through analyzing millions of videos across various domains. To further enhance WorldGPT's capability in specialized scenarios and long-term tasks, we have integrated it with a novel cognitive architecture that combines memory offloading, knowledge retrieval, and context reflection. As for evaluation, we build WorldNet, a multimodal state transition prediction benchmark encompassing varied real-life scenarios. Conducting evaluations on WorldNet directly demonstrates WorldGPT's capability to accurately model state transition patterns, affirming its effectiveness in understanding and predicting the dynamics of complex scenarios. We further explore WorldGPT's emerging potential in serving as a world simulator, helping multimodal agents generalize to unfamiliar domains through efficiently synthesising multimodal instruction instances which are proved to be as reliable as authentic data for fine-tuning purposes. The project is available on \url{https://github.com/DCDmllm/WorldGPT}.
Paper Structure (19 sections, 7 figures, 6 tables)

This paper contains 19 sections, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. WorldGPT
  4. MLLM as Foundation of World Model
  5. Progressively State Transition Training
  6. Cognitive Architecture
  7. Details of the Cognitive Architecture
  8. Cognitive-Augmented Tuning
  9. WorldNet
  10. Constructing WorldNet-Wild from Unlabelled Videos
  11. Constructing WorldNet-Crafted from Various Sources.
  12. Applying WorldGPT as a Multimodal Instruction Synthesizer
  13. Textual Instruction Generation
  14. Multimodal Instruction Completion
  15. Filtering and Post-processing
  16. ...and 4 more sections

Figures (7)

  • Figure 1: (Left) Progressively pretraining stage. (Right) Cognitive-augmented tuning stage.
  • Figure 2: Loss for three types of tasks during progressively state transition training.
  • Figure 3: Loss for three types of tasks during naive state transition training.
  • Figure 4: (Left) The working flow of WorldGPT and cognitive architecture. (Right) The detailed model architecture of ContextReflector.
  • Figure 5: Representative cases selected from WorldNet. WorldNet contains state transition samples across diverse domains.
  • ...and 2 more figures