Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directions

TL;DR

This paper surveys Embodied Multimodal Large Models (EMLMs), detailing how large language, vision, and cross-modal models are being integrated to enable perception, reasoning, and action in real-world environments. It provides a comprehensive, full-stack analysis of foundational models, embodied perception/navigation/interaction/simulation components, and the diverse datasets that train and evaluate these systems, drawing on hundreds of recent works. The authors identify core challenges—cross-modal alignment, scalability, generalization, and data quality—and propose directions such as cross-modal pre-training, self-supervised learning, and end-to-end large models to propel the field forward. By synthesizing state-of-the-art methods and benchmarks, the paper aims to guide future research and practical deployments in autonomous robotics and intelligent agents while highlighting ethical considerations inherent to embodied AI.

Abstract

Embodied multimodal large models (EMLMs) have gained significant attention in recent years due to their potential to bridge the gap between perception, cognition, and action in complex, real-world environments. This comprehensive review explores the development of such models, including Large Language Models (LLMs), Large Vision Models (LVMs), and other models, while also examining other emerging architectures. We discuss the evolution of EMLMs, with a focus on embodied perception, navigation, interaction, and simulation. Furthermore, the review provides a detailed analysis of the datasets used for training and evaluating these models, highlighting the importance of diverse, high-quality data for effective learning. The paper also identifies key challenges faced by EMLMs, including issues of scalability, generalization, and real-time decision-making. Finally, we outline future directions, emphasizing the integration of multimodal sensing, reasoning, and action to advance the development of increasingly autonomous systems. By providing an in-depth analysis of state-of-the-art methods and identifying critical gaps, this paper aims to inspire future advancements in EMLMs and their applications across diverse domains.

Figures (6)

• Figure 1: A timeline of research progress in the field of Embodied Perception, Navigation and Interaction.
• Figure 2: Examples of embodied AI agents.
• Figure 3: Full Task Stacks for Embodied Agents. Various embodied intelligent agents, including robot dogs, humanoid robots, and other types of intelligent systems, rely on a range of sensors, such as cameras, LiDAR, and other sensing technologies, to perceive their environment. These agents then perform specific tasks, usually guided by human voice or language commands. Task execution typically involves three key modules: perception, navigation, and interaction. The datasets and large models required for these modules can be collected and trained using either simulators or real-world scenarios. During task execution, the agent interacts with its environment to gather the necessary information.
• Figure 4: Diagram illustrating the Rekep framework (Source: Rekep huang2024rekep).
• Figure 5: Diagram illustrating the NavGPT-2 framework (Source: NavGPT-2 zhou2025navgpt).