Grounding World Simulation Models in a Real-World Metropolis

Abstract

What if a world simulation model could render not an imagined environment but a city that actually exists? Prior generative world models synthesize visually plausible yet artificial environments by imagining all content. We present Seoul World Model (SWM), a city-scale world model grounded in the real city of Seoul. SWM anchors autoregressive video generation through retrieval-augmented conditioning on nearby street-view images. However, this design introduces several challenges, including temporal misalignment between retrieved references and the dynamic target scene, limited trajectory diversity and data sparsity from vehicle-mounted captures at sparse intervals. We address these challenges through cross-temporal pairing, a large-scale synthetic dataset enabling diverse camera trajectories, and a view interpolation pipeline that synthesizes coherent training videos from sparse street-view images. We further introduce a Virtual Lookahead Sink to stabilize long-horizon generation by continuously re-grounding each chunk to a retrieved image at a future location. We evaluate SWM against recent video world models across three cities: Seoul, Busan, and Ann Arbor. SWM outperforms existing methods in generating spatially faithful, temporally consistent, long-horizon videos grounded in actual urban environments over trajectories reaching hundreds of meters, while supporting diverse camera movements and text-prompted scenario variations.

Figures (16)

• Figure 1: Seoul World Model (SWM) generates videos over a kilometer grounded in a real city. A camera trajectory placed on a map produces continuous dynamic video depicting actual surroundings along the route. Users can further reshape the scene through text prompts, enabling imaginative scenarios.
• Figure 2: Data overview: (a) real street-view and (b) synthetic datasets. The satellite-view map (leftmost column) shows the target video trajectory, with green and red indicating the start and end points, and yellow indicating reference view locations selected via cross-temporal pairing.
• Figure 3: View interpolation pipeline: (a) Keyframe conditioning via channel concatenation, and (b) Keyframe conditioning with intermittent freeze-frame strategy (Ours). 4F and 1F denote latents derived from four frames and one frame, respectively, prior to the 4$\times$ temporal compression of the 3D VAE.
• Figure 4: Model overview. Given a start location, SWM autoregressively generates video grounded in a real city based on text prompt $P^{(i)}$, and target camera trajectory $\mathbf{C}^{(i)}$, retrieving the relevant street-view images from a geo-indexed database. These retrieved images provide a Virtual Lookahead Sink for long-horizon stability and serve as conditioning for geometric referencing and semantic referencing to ground the generation in real-world geometry and appearance. The model generates each chunk autoregressively conditioned on self-generated history.
• Figure 5: Virtual Lookahead Sink: (a) Vanilla attention sink, (b) virtual lookahead sink (Ours). (a) anchors to the initial frame, whose guidance weakens as the camera moves farther away. (b) dynamically retrieves the nearest street-view as a virtual future destination.