UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videos

TL;DR

UrbanVerse tackles the scalability challenge of training urban embodied AI by turning free-form city-tour videos into physics-enabled simulation scenes. It introduces UrbanVerse-100K, a large, annotated asset database, and UrbanVerse-Gen, a real-to-sim pipeline that distills videos into scene graphs and instantiates diverse, interaction-ready scenes in IsaacSim. The work provides a 160-scene library spanning 24 countries and a CraftBench for evaluation, demonstrating that policies trained in UrbanVerse scale in generalization via power-law behavior and transfer effectively to real-world deployments, including zero-shot sim-to-real scenarios. With high reconstruction fidelity against KITTI-360 and robust real-world performance across two robot embodiments, UrbanVerse offers a practical, open-source pathway to realistic, scalable urban simulation for embodied AI research.

Abstract

Urban embodied AI agents, ranging from delivery robots to quadrupeds, are increasingly populating our cities, navigating chaotic streets to provide last-mile connectivity. Training such agents requires diverse, high-fidelity urban environments to scale, yet existing human-crafted or procedurally generated simulation scenes either lack scalability or fail to capture real-world complexity. We introduce UrbanVerse, a data-driven real-to-sim system that converts crowd-sourced city-tour videos into physics-aware, interactive simulation scenes. UrbanVerse consists of: (i) UrbanVerse-100K, a repository of 100k+ annotated urban 3D assets with semantic and physical attributes, and (ii) UrbanVerse-Gen, an automatic pipeline that extracts scene layouts from video and instantiates metric-scale 3D simulations using retrieved assets. Running in IsaacSim, UrbanVerse offers 160 high-quality constructed scenes from 24 countries, along with a curated benchmark of 10 artist-designed test scenes. Experiments show that UrbanVerse scenes preserve real-world semantics and layouts, achieving human-evaluated realism comparable to manually crafted scenes. In urban navigation, policies trained in UrbanVerse exhibit scaling power laws and strong generalization, improving success by +6.3% in simulation and +30.1% in zero-shot sim-to-real transfer comparing to prior methods, accomplishing a 300 m real-world mission with only two interventions.

Figures (33)

• Figure 1: UrbanVerse system converts real-world urban scenes from city-tour videos into physics-aware, interactive simulation environments, enabling scalable robot learning in urban spaces with real-world generalization.
• Figure 2: Example instances from our large-scale urban asset database UrbanVerse-100K. Assets range from a 0.03 m crushed can to a 200 m skyscraper, all annotated to metric scale; note the realistic relative scales between objects, with road and sidewalk materials and sky maps ensuring realistic ground appearance and illumination.
• Figure 3: Example of annotated object attributes.
• Figure 3: Real-world results.
• Figure 4: UrbanVerse-Gen pipeline:(1) Given an RGB city-tour video as input, it distills object semantics, appearances, and 3D layouts, along with ground composition and sky illumination representations, into a unified scene graph containing object, ground, and sky nodes; (2) Based on the distilled representations, it retrieves multiple matched assets from the UrbanVerse-100K for each node using using diverse similarity measures; (3) By progressively assembling one matched asset per node, it generates a complete digital cousin simulation scene.