UrbanGraphEmbeddings: Learning and Evaluating Spatially Grounded Multimodal Embeddings for Urban Science
Jie Zhang, Xingtong Yu, Yuan Fang, Rudi Stouffs, Zdravko Trivic
TL;DR
UrbanGraphEmbedding addresses the need for spatial grounding in urban vision–language models by introducing UGData, which anchors street-view imagery to spatial graphs to expose distance, direction, connectivity, and neighborhood context. It then proposes a two-stage training framework, UrbanGraphEmbedding (UGE), combining instruction-guided contrastive learning (Stage 1) with a graph-conditioned spatial encoder (Stage 2) to align images, text, and spatial structures. The approach leverages a graph encoder with node and edge features, including sinusoidal spatial encodings and distance/direction features, and uses LoRA-tuned fixed-dimensional spatial embeddings. Evaluations on UGBench across geolocation ranking, image retrieval, urban perception, and spatial grounding show substantial gains, e.g., up to $44\%$ improvement in image retrieval and $30\%$ in geolocation ranking on training cities, and over $30\%$ and $22\%$ gains on held-out cities, demonstrating the value of explicit spatial grounding. It highlights practical place-centric AI applications and points to future directions in improving fine-grained metric spatial reasoning and broader deployment opportunities.
Abstract
Learning transferable multimodal embeddings for urban environments is challenging because urban understanding is inherently spatial, yet existing datasets and benchmarks lack explicit alignment between street-view images and urban structure. We introduce UGData, a spatially grounded dataset that anchors street-view images to structured spatial graphs and provides graph-aligned supervision via spatial reasoning paths and spatial context captions, exposing distance, directionality, connectivity, and neighborhood context beyond image content. Building on UGData, we propose UGE, a two-stage training strategy that progressively and stably aligns images, text, and spatial structures by combining instruction-guided contrastive learning with graph-based spatial encoding. We finally introduce UGBench, a comprehensive benchmark to evaluate how spatially grounded embeddings support diverse urban understanding tasks -- including geolocation ranking, image retrieval, urban perception, and spatial grounding. We develop UGE on multiple state-of-the-art VLM backbones, including Qwen2-VL, Qwen2.5-VL, Phi-3-Vision, and LLaVA1.6-Mistral, and train fixed-dimensional spatial embeddings with LoRA tuning. UGE built upon Qwen2.5-VL-7B backbone achieves up to 44% improvement in image retrieval and 30% in geolocation ranking on training cities, and over 30% and 22% gains respectively on held-out cities, demonstrating the effectiveness of explicit spatial grounding for spatially intensive urban tasks.