UrbanFusion: Stochastic Multimodal Fusion for Contrastive Learning of Robust Spatial Representations

TL;DR

UrbanFusion tackles the challenge of learning robust, transferable spatial representations from heterogeneous geospatial data by introducing Stochastic Multimodal Fusion (SMF), a framework that jointly leverages modality-specific encoders and a Transformer fusion to integrate street view, remote sensing, basemaps, and POIs. It combines a contrastive location alignment objective with latent modality reconstruction, enabling flexible inference with any subset of modalities and preserving redundant, unique, and synergistic information. A large-scale PP2-M pretraining setup across 56 cities, coupled with evaluation on 41 downstream tasks, demonstrates superior cross-modal and cross-regional generalization compared to state-of-the-art GeoFMs, even under incomplete data. This work offers a scalable, data-efficient path to foundation-level geospatial representations with broad applicability in urban analytics and beyond.

Abstract

Forecasting urban phenomena such as housing prices and public health indicators requires the effective integration of various geospatial data. Current methods primarily utilize task-specific models, while recent foundation models for spatial representations often support only limited modalities and lack multimodal fusion capabilities. To overcome these challenges, we present UrbanFusion, a Geo-Foundation Model (GeoFM) that features Stochastic Multimodal Fusion (SMF). The framework employs modality-specific encoders to process different types of inputs, including street view imagery, remote sensing data, cartographic maps, and points of interest (POIs) data. These multimodal inputs are integrated via a Transformer-based fusion module that learns unified representations. An extensive evaluation across 41 tasks in 56 cities worldwide demonstrates UrbanFusion's strong generalization and predictive performance compared to state-of-the-art GeoAI models. Specifically, it 1) outperforms prior foundation models on location-encoding, 2) allows multimodal input during inference, and 3) generalizes well to regions unseen during training. UrbanFusion can flexibly utilize any subset of available modalities for a given location during both pretraining and inference, enabling broad applicability across diverse data availability scenarios. All source code is available at https://github.com/DominikM198/UrbanFusion.

Figures (12)

• Figure 1: UrbanFusion. Pretrained modality-specific encoders extract features projected into tokens. After random token masking, a Transformer fuses the tokens. The output feeds into two heads: one for Contrastive Location Alignment (CL), the other for Latent Modality Reconstruction (Rec.). For downstream tasks, coordinates or available modalities are input into the frozen encoder (green arrows) to obtain feature vectors for training downstream models.
• Figure 2: Visual comparison of Multimodal location embeddings, all trained on the PP2-M dataset. Embeddings are grouped into 10 clusters using $k$-means.
• Figure 3: Empirical results on synthetic data analyzing the preservation of redundant, unique, and synergistic information using the Partial Information Decomposition (PID) framework.
• Figure 4: RGB composite image of the top three principal components of location representations computed globally for the Portland area using only coordinates. (part 1 of 2)
• Figure 5: RGB composite image of the top three principal components of location representations computed globally for the Portland area using only coordinates. (part 2 of 2)