Embodied AI Agents: Modeling the World

TL;DR

The paper argues that embodied AI agents require integrated world models that fuse multimodal perception, physical reasoning, and mental models of users to plan and act autonomously. It surveys three agent types—virtual embodied, wearable, and robotic—and details architectures, benchmarks, and memory mechanisms that enable efficient, long-horizon decision making. A central theme is the development of scalable, lifelong learning through systems that couple observation-driven and action-driven paradigms, including multi-agent collaboration. The work also foregrounds ethical considerations such as privacy and anthropomorphism, outlining directions for social intelligence, safe deployment, and responsible design. Together, these components delineate a roadmap for building capable, interactive, and trustworthy embodied AI that can operate across digital and physical environments.

Abstract

This paper describes our research on AI agents embodied in visual, virtual or physical forms, enabling them to interact with both users and their environments. These agents, which include virtual avatars, wearable devices, and robots, are designed to perceive, learn and act within their surroundings, which makes them more similar to how humans learn and interact with the environments as compared to disembodied agents. We propose that the development of world models is central to reasoning and planning of embodied AI agents, allowing these agents to understand and predict their environment, to understand user intentions and social contexts, thereby enhancing their ability to perform complex tasks autonomously. World modeling encompasses the integration of multimodal perception, planning through reasoning for action and control, and memory to create a comprehensive understanding of the physical world. Beyond the physical world, we also propose to learn the mental world model of users to enable better human-agent collaboration.

Figures (17)

• Figure 1: Overview of the Embodied AI Agent architecture, with the interaction loop between user, world and agent. The world model is the core component responsible for planning and reasoning.
• Figure 2: The modular system architecture for autonomous intelligence proposed by PathAMI, illustrating its interaction with the world. Embodied AI agents introduce an additional dimension of interaction with the user.
• Figure 3: Planning with a JEPA visual world model for a robotic manipulation task. The future is imagined by encoding the context frames with the encoder $E_{\theta}$ and rolling out the predictor $P_{\phi}$ from these context visual embeddings and actions. To plan, any cost function with respect to the goal can be used. In this figure, the $L_1$ distance of the last imagined state to the goal state embedding is optimized with respect to the actions $(a_k)_{k \in [T]}$ using any trajectory optimization method, e.g. rubinstein1997optimization. Taken from VJEPA2.
• Figure 4: Comparison of action ($a$) conditioned world model architectures. VLWM is a JEPA-style world model that predict abstract representation of future world states, instead of generating noisy and high-volume raw observations.
• Figure 5: Probing nested beliefs through program guided scenarios. We adversarially generate belief reasoning tasks via ExploreTom exploretom2024.