Virtual Community: An Open World for Humans, Robots, and Society

TL;DR

Virtual Community introduces an open‑world, unified simulation platform for humans and robots built atop the Genesis physics engine to study embodied social intelligence at scale. It combines a real‑world geospatial data–driven open world generator with a generative pipeline for scene and agent creation, including LLM‑driven grounded character profiles and schedules. The work defines two open‑world benchmarks—the Community Planning Challenge and the Community Robot Challenge—for evaluating high‑level multi‑agent planning and low‑level robotic cooperation, respectively, across indoor and outdoor settings. Results across baselines and ablations demonstrate both the promise of unified human–robot embodied AI in open worlds and the current gaps in planning under uncertainty, visual fidelity, and scalable agent populations, pointing to future improvements in realism, scale, and rendering.

Abstract

The rapid progress in AI and Robotics may lead to a profound societal transformation, as humans and robots begin to coexist within shared communities, introducing both opportunities and challenges. To explore this future, we present Virtual Community-an open-world platform for humans, robots, and society-built on a universal physics engine and grounded in real-world 3D scenes. With Virtual Community, we aim to enable the study of embodied social intelligence at scale. To support these, Virtual Community features: 1) An open-source multi-agent physics simulator that supports robots, humans, and their interactions within a society; 2) A large-scale, real-world aligned community generation pipeline, including vast outdoor space, diverse indoor scenes, and a community of grounded agents with rich characters and appearances. Leveraging Virtual Community, we propose two novel challenges. The Community Planning Challenge evaluates multi-agent reasoning and planning ability in open-world settings, such as cooperating to help agents with daily activities and efficiently connecting other agents. The Community Robot Challenge requires multiple heterogeneous robots to collaborate in solving complex open-world tasks. We evaluate various baselines on these tasks and demonstrate the challenges in both high-level open-world task planning and low-level cooperation controls. We hope that Virtual Community will unlock further study of human-robot coexistence within open-world environments.

Figures (18)

• Figure 1: Virtual Community supports embodied multi‑agent simulation in open‑world environments. We introduce an automated pipeline that generates open-world scenes and agent communities, with agents instantiated as humanoid avatars or robots to enable diverse social interactions.
• Figure 2: Framework of the Virtual Community Generation Pipeline. This pipeline generates scenes and corresponding human agents from real-world geospatial data. The scene generation component (A) refines rough 3D data by using generative models to enhance textures and geospatial data to simplify geometry. It also utilizes generative methods to create interactive objects and detailed indoor scenes. The agent generation component (B) leverages LLMs to generate agent characters and social relationship networks based on scene descriptions. (C) We simulate the human communities and robots in the open world scenes based on Genesis engine.
• Figure 3: Egocentric view of the generated scenes. The resulting scene features clean geometry and realistic textures, which support physical simulation and enhance real-world style fidelity.
• Figure 4: We evaluate baseline agents on the Community Influence Task across five communities. Results show that more powerful LLMs are better able to connect with and influence other members in the community.
• Figure 5: We decompose the outdoor 3D scene into three components - the terrain, buildings, and roofs. Each part is generated separately using different strategies to balance the visual appearance and geometry complexity.