A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI

TL;DR

This survey addresses the challenge of training navigation and manipulation agents for Embodied AI by evaluating the role of physics-based simulators in mitigating the sim-to-real gap. It synthesizes simulator properties, benchmark datasets, evaluation metrics, and cutting-edge methods including differentiable physics, world models, and vision-language-action frameworks. The authors propose a structured resource that helps researchers choose appropriate tools while considering hardware constraints, and they identify future directions such as efficient and continual learning, equivariant representations, and advanced evaluation paradigms. Overall, the work highlights a shift from purely model-based learning toward data-rich, multimodal, and differentiable approaches that improve transferability to real robots and real-world settings.

Abstract

Navigation and manipulation are core capabilities in Embodied AI, yet training agents with these capabilities in the real world faces high costs and time complexity. Therefore, sim-to-real transfer has emerged as a key approach, yet the sim-to-real gap persists. This survey examines how physics simulators address this gap by analyzing their properties overlooked in previous surveys. We also analyze their features for navigation and manipulation tasks, along with hardware requirements. Additionally, we offer a resource with benchmark datasets, metrics, simulation platforms, and cutting-edge methods-such as world models and geometric equivariance-to help researchers select suitable tools while accounting for hardware constraints.

Figures (10)

• Figure 1: Timeline illustrating the evolution of navigation (top) and manipulation (bottom) research in Embodied AI from 2019 onward. It highlights key methodologies---including Explicit and Latent memory, Foundation Model, World Model, Reinforcement Learning (RL), Imitation Learning (IL), Diffusion Policy (DP), and Vision-Language-Action (VLA) approaches---and benchmark datasets. Benchmarks provide a foundation for training and evaluating agents, and the introduction of new benchmarks often inspires innovative methods, which together advance the field.
• Figure 2: A taxonomy of this survey, focusing on two main tasks of Embodied AI: Navigation and Manipulation. We discuss the components of each task, including tasks, simulators, datasets, evaluation metrics, and methods. For each component, we further divide into different types to structure the analysis.
• Figure 3: This diagram outlines the four key steps in navigation tasks---Perception, Memory Building, Decision-Making, and Action Execution---along with the two sim-to-real challenges of visual rendering and physical dynamics. Navigation tasks are categorized into goal-driven (e.g., PointNav, ImageNav, ObjectNav) and task-driven (e.g., EQA embodiedqa, VLN mattersim). The memory can be categorized into explicit memory and implicit memory.
• Figure 4: Illustration of navigation tasks in embodied agents. (a) The first-person camera view of the robot in an indoor environment, highlighting partial observability challenges. (b) An occupancy map used for planning and model building, showing free (white) and occupied (gray) spaces with a goal pose (green arrow). (c) A simulated environment with a robotic agent navigating a hallway.
• Figure 5: This figure showcases six navigation simulators, categorized into indoor, outdoor, and general-purpose types, arranged from left to right. The first two, Habitat and AI2-THOR, represent indoor settings: Habitat simulates navigation in a realistic 3D kitchen scene with a top-down occupancy map, while AI2-THOR simulates a wheeled robot navigating and interacting with objects, such as a tomato, in a home environment. The next two, CARLA and AirSim, illustrate open-world scenarios in outdoor settings: CARLA displays a car on an urban road with traffic, and AirSim simulates a drone navigating a suburban street with a depth sensor view. The final two, ThreeDWorld and Isaac Sim, represent general-purpose settings: ThreeDWorld displays a wheeled robot navigating in a photorealistic bedroom environment, and Isaac Sim shows a mobile robot navigating inside a warehouse environment.