GRUtopia: Dream General Robots in a City at Scale

TL;DR

GRUtopia proposes a scalable Sim2Real platform for embodied AI by uniting GRScenes (100k city-scale scenes across 89 categories), GRResidents (LLM-driven NPCs with a World Knowledge Manager), and GRBench (three benchmarks for object navigation, social interaction, and loco-manipulation). The framework enables rich social and physical interactions, leveraging RL-based robot controls and multi-modal reasoning to push beyond home-focused environments. Quantitative and qualitative results show improved performance with advanced backends, while also highlighting persistent challenges in loco-manipulation and transfer to real-world-like settings. This platform provides a comprehensive, extensible foundation for evaluating and advancing embodied agents in large-scale, diverse environments.

Abstract

Recent works have been exploring the scaling laws in the field of Embodied AI. Given the prohibitive costs of collecting real-world data, we believe the Simulation-to-Real (Sim2Real) paradigm is a crucial step for scaling the learning of embodied models. This paper introduces project GRUtopia, the first simulated interactive 3D society designed for various robots. It features several advancements: (a) The scene dataset, GRScenes, includes 100k interactive, finely annotated scenes, which can be freely combined into city-scale environments. In contrast to previous works mainly focusing on home, GRScenes covers 89 diverse scene categories, bridging the gap of service-oriented environments where general robots would be initially deployed. (b) GRResidents, a Large Language Model (LLM) driven Non-Player Character (NPC) system that is responsible for social interaction, task generation, and task assignment, thus simulating social scenarios for embodied AI applications. (c) The benchmark, GRBench, supports various robots but focuses on legged robots as primary agents and poses moderately challenging tasks involving Object Loco-Navigation, Social Loco-Navigation, and Loco-Manipulation. We hope that this work can alleviate the scarcity of high-quality data in this field and provide a more comprehensive assessment of Embodied AI research. The project is available at https://github.com/OpenRobotLab/GRUtopia.

Figures (11)

• Figure 1: Key features of GRUtopia.
• Figure 2: The richness and diversity of scenes and objects in GRScenes. (a) The distribution of scenes is shown in a fine granularity. GRScenes covers a vast range of functional scene categories. (b) The spectrum of annotated objects and some examples present in GRScenes.
• Figure 2: Cross-verification (Referring & Grounding) accuracy (%) and QA score of our NPCs driven by different LLMs.
• Figure 3: Overview of GRResidents (Sec. \ref{['sec: generative npc in the wild']}). It comprises two modules: (a) A world knowledge manager that organizes scene knowledge from dataset annotations and simulator backend and provides APIs for knowledge retrieving. (b) An LLM planner that is able to retrieve global knowledge from the world knowledge manager and generate responses according to dialogue context.
• Figure 4: Evaluation of the NPC's capabilities in description and grounding through cross-verification between NPC and human.