EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment

TL;DR

EmbodiedCity delivers a high-fidelity urban embodied AI benchmark built on a real Beijing district, enabling continuous, multi-agent interactions and a diverse task suite spanning perception, reasoning, and decision-making. It provides a full simulation-stack (Unreal Engine 5.3, AirSim-based sensing, Mirage dynamics), an open Python SDK, and an online platform, plus a large-scale dataset of 87.1k cases across five embodied tasks. The paper demonstrates the platform's utility by evaluating multiple multi-modal LLMs, revealing strengths and limitations across tasks and languages, and discussing sim2real implications. This work aims to narrow the indoor-bias in embodied AI and push toward robust urban-embodied intelligence with real-world impact.

Abstract

Embodied artificial intelligence emphasizes the role of an agent's body in generating human-like behaviors. The recent efforts on EmbodiedAI pay a lot of attention to building up machine learning models to possess perceiving, planning, and acting abilities, thereby enabling real-time interaction with the world. However, most works focus on bounded indoor environments, such as navigation in a room or manipulating a device, with limited exploration of embodying the agents in open-world scenarios. That is, embodied intelligence in the open and outdoor environment is less explored, for which one potential reason is the lack of high-quality simulators, benchmarks, and datasets. To address it, in this paper, we construct a benchmark platform for embodied intelligence evaluation in real-world city environments. Specifically, we first construct a highly realistic 3D simulation environment based on the real buildings, roads, and other elements in a real city. In this environment, we combine historically collected data and simulation algorithms to conduct simulations of pedestrian and vehicle flows with high fidelity. Further, we designed a set of evaluation tasks covering different EmbodiedAI abilities. Moreover, we provide a complete set of input and output interfaces for access, enabling embodied agents to easily take task requirements and current environmental observations as input and then make decisions and obtain performance evaluations. On the one hand, it expands the capability of existing embodied intelligence to higher levels. On the other hand, it has a higher practical value in the real world and can support more potential applications for artificial general intelligence. Based on this platform, we evaluate some popular large language models for embodied intelligence capabilities of different dimensions and difficulties.

Figures (13)

• Figure 1: An embodied environment built upon real cities includes realistic urban landscapes such as streets, buildings, city elements, pedestrians, and traffic. It also offers interaction interfaces for aerial and ground agents with the urban environment.
• Figure 2: The framework of our platform, including simulator, interface, and embodied tasks.
• Figure 3: The image showcases various components of our city simulator.
• Figure 4: Integration of agent interface within the Unreal Engine city simulator environment, showcasing the simulation of both drones and unmanned vehicles (cars).
• Figure 5: Illustration of the embodied tasks in the urban environment.