GAIR: Improving Multimodal Geo-Foundation Model with Geo-Aligned Implicit Representations

TL;DR

GAIR tackles the challenge of limited labeled geospatial data by fusing overhead remote sensing, street-view imagery, and geolocation through a factorized encoder and an implicit neural representation module. It introduces two contrastive objectives, INCL and SECL, to learn geo-aligned embeddings across modalities, with INR refining RS representations to local coordinates. Pretrained on Streetscape1M, GAIR demonstrates state-of-the-art performance across 10 geospatial tasks spanning street-view, remote-sensing, and location benchmarks, highlighting strong generalization and transferability. The work emphasizes the importance of geo-alignment across modalities and validates the utility of location priors in improving geospatial understanding for applications from socio-economic sensing to species recognition.

Abstract

Advancements in vision and language foundation models have inspired the development of geo-foundation models (GeoFMs), enhancing performance across diverse geospatial tasks. However, many existing GeoFMs primarily focus on overhead remote sensing (RS) data while neglecting other data modalities such as ground-level imagery. A key challenge in multimodal GeoFM development is to explicitly model geospatial relationships across modalities, which enables generalizability across tasks, spatial scales, and temporal contexts. To address these limitations, we propose GAIR, a novel multimodal GeoFM architecture integrating overhead RS data, street view (SV) imagery, and their geolocation metadata. We utilize three factorized neural encoders to project an SV image, its geolocation, and an RS image into the embedding space. The SV image needs to be located within the RS image's spatial footprint but does not need to be at its geographic center. In order to geographically align the SV image and RS image, we propose a novel implicit neural representations (INR) module that learns a continuous RS image representation and looks up the RS embedding at the SV image's geolocation. Next, these geographically aligned SV embedding, RS embedding, and location embedding are trained with contrastive learning objectives from unlabeled data. We evaluate GAIR across 10 geospatial tasks spanning RS image-based, SV image-based, and location embedding-based benchmarks. Experimental results demonstrate that GAIR outperforms state-of-the-art GeoFMs and other strong baselines, highlighting its effectiveness in learning generalizable and transferable geospatial representations.

Figures (10)

• Figure 1: Overview of the proposed GAIR architecture, which encodes data of three geospatial modalities: street view image $s_i$, its geolocation coordinate $x_i$, and remote sensing image $r_i$,. The location encoder$e$ is an implicit function transforming geolocation $x_i$ into a latent representation $e(x_i)$, while Vision Transformer (ViT)-based image encoders extract representations from remote sensing and street view images, denoted as $f(r_i)$ and $g(s_i)$, respectively. A novel Implicit Neural Representation (INR) module refines $f(r_i)$ into a localized embedding $z_{i}^q$, which is geographically aligned with $g(s_i)$ and $e(x_i)$. $z_{i}$, $g(s_i)$, and $e(x_i)$ are trained in a self-supervised manner through contrastive learning.
• Figure 2: Overview of the proposed GAIR architecture, which encodes data of three geospatial modalities: geolocation coordinate $x_i$, remote sensing image $r^t_i$, and street view image $s_i$. The location encoder $e$ transforms geolocation $x_i$ into a latent representation $e(x_i)$, while ViT-based image encoders extract representations from remote sensing and street view images, denoted as $f(r_i)$ and $g(s_i)$, respectively. Implicit Neural Representation (INR) refines $f(r_i)$ into a localized embedding $z_{i}^{(q)}$ through feature unfolding and local ensemble interpolation. Then we do contrastive learning to align these three feature embeddings.
• Figure 3: The model evaluation pipelines on three benchmarks covering 10 tasks. Specifically, after pretraining GAIR, we fine-tune the street view image encoder $g$ to tackle the street view imagery benchmark, the RS image encoder $f$ for remote sensing imagery benchamrk, and the location encoder $e$ for location benchmark.
• Figure 4: Visualization of spatial alignment across different modalities in GAIR using heat maps. The red star indicates the geographic location of the street view (SV) image $s_i$. (a) Cosine similarity between the SV image embedding $g(s_i)$ and the localized remote sensing (RS) image embeddings $\hat{r}_i(x)$. (b) Cosine similarity between $g(s_i)$ and the location embeddings $e(x_i)$. Spatial grids are uniformly sampled across the Greater London region to generate location embeddings. More visualizations can be seen in Appendix B.8
• Figure 5: An illustration on the pipeline to construct our StreetscapeIM GeoFM pretraining dataset.