From Efficient Multimodal Models to World Models: A Survey

TL;DR

This survey analyzes the state of Multimodal Large Models (MLMs) and their potential to evolve into world models by enabling cross-modal understanding and generation. It surveys core MLM techniques such as Multimodal Chain of Thought (M-COT), Multimodal Instruction Tuning (M-IT), and Multimodal In-Context Learning (M-ICL), and discusses the role of 3D generation and embodied intelligence in world simulators. It contrasts autoregressive approaches with JEPA-based hierarchical planning as paths to world models, and highlights challenges in unifying a single model, efficiency bottlenecks, and the integration of external rule systems. The paper articulates practical directions—data-driven scaling, rule-based grounding, and computational optimizations—that are essential for advancing toward robust, embodied, and capable world simulators in real-world settings.

Abstract

Multimodal Large Models (MLMs) are becoming a significant research focus, combining powerful large language models with multimodal learning to perform complex tasks across different data modalities. This review explores the latest developments and challenges in MLMs, emphasizing their potential in achieving artificial general intelligence and as a pathway to world models. We provide an overview of key techniques such as Multimodal Chain of Thought (M-COT), Multimodal Instruction Tuning (M-IT), and Multimodal In-Context Learning (M-ICL). Additionally, we discuss both the fundamental and specific technologies of multimodal models, highlighting their applications, input/output modalities, and design characteristics. Despite significant advancements, the development of a unified multimodal model remains elusive. We discuss the integration of 3D generation and embodied intelligence to enhance world simulation capabilities and propose incorporating external rule systems for improved reasoning and decision-making. Finally, we outline future research directions to address these challenges and advance the field.

Figures (8)

• Figure 1: The agent inputs actions into the world simulator, which changes states and outputs feedback. This interaction loop illustrates the process where the agent perceives the current situation, makes decisions, and acts. The world simulator updates its state based on these actions and provides feedback to the agent, essential for learning and adaptation.
• Figure 2: Core principle of simplifying calculations in linear attention mechanisms. The traditional self-attention mechanism with $O(N^2)$ complexity (top) is replaced by a more efficient linear attention approach with $O(N)$ complexity (bottom). This is achieved by removing the softmax operation and approximating the function, allowing attention to be computed as a series of matrix multiplications that scale linearly with input size.
• Figure 3: Illustration of Multi-Head Attention (MHA), Multi-Query Attention (MQA), and Group-Query Attention (GQA). This figure demonstrates the structural differences between these attention mechanisms. Multi-Head Attention uses multiple sets of queries, keys, and values to capture different aspects of the input. Grouped-Query Attention shares keys and values among groups of queries to balance computational efficiency and model expressiveness. Multi-Query Attention simplifies the model by sharing the keys and values across all attention heads, reducing the computational load while maintaining performance.
• Figure 4: Architectural differences between SSM and Transformer in processing multidimensional inputs. The SSM (Selective State Space Model) processes each input dimension independently, allowing for high parallel computation and linear complexity. In contrast, the Transformer uses a multi-head attention mechanism to capture global dependencies across input dimensions, resulting in higher computational complexity but more comprehensive contextual understanding.
• Figure 5: H3 structure compared with SSM. The figure shows different architectures like H3, Gated MLP, S6, and SSD, explaining how each configuration incorporates selective state space models (SSM) and convolutional layers to handle inputs and generate outputs efficiently.