GeoToken: Hierarchical Geolocalization of Images via Next Token Prediction

TL;DR

GeoToken reframes worldwide image geolocalization as coarse-to-fine autoregressive token prediction over a hierarchical S2 grid, predicting a sequence of geographic tokens conditioned on an image and retrieved context. It integrates retrieval-augmented context, multi-modal geo-alignment encoders, and test-time sampling strategies to generate multiple plausible location hypotheses, then reranks them via log-probability, similarity, rewards, or an MLLM judge. Across MLMM-free and MLMM-assisted settings, GeoToken achieves state-of-the-art results on IM2GPS3K and YFCC4K, with notable privacy advantages due to on-device inference. The approach offers a flexible, uncertainty-aware framework that can be extended with additional modalities and retrieval schemes while maintaining strong performance and interpretability in geolocalization tasks.

Abstract

Image geolocalization, the task of determining an image's geographic origin, poses significant challenges, largely due to visual similarities across disparate locations and the large search space. To address these issues, we propose a hierarchical sequence prediction approach inspired by how humans narrow down locations from broad regions to specific addresses. Analogously, our model predicts geographic tokens hierarchically, first identifying a general region and then sequentially refining predictions to increasingly precise locations. Rather than relying on explicit semantic partitions, our method uses S2 cells, a nested, multiresolution global grid, and sequentially predicts finer-level cells conditioned on visual inputs and previous predictions. This procedure mirrors autoregressive text generation in large language models. Much like in language modeling, final performance depends not only on training but also on inference-time strategy. We investigate multiple top-down traversal methods for autoregressive sampling, incorporating techniques from test-time compute scaling used in language models. Specifically, we integrate beam search and multi-sample inference while exploring various selection strategies to determine the final output. This enables the model to manage uncertainty by exploring multiple plausible paths through the hierarchy. We evaluate our method on the Im2GPS3k and YFCC4k datasets against two distinct sets of baselines: those that operate without a Multimodal Large Language Model (MLLM) and those that leverage one. In the MLLM-free setting, our model surpasses other comparable baselines on nearly all metrics, achieving state-of-the-art performance with accuracy gains of up to 13.9%. When augmented with an MLLM, our model outperforms all baselines, setting a new state-of-the-art across all metrics. The source code is available at https://github.com/NNargesNN/GeoToken.

Figures (4)

• Figure 1: (a) Distribution of the MP16 dataset, with over 4 million samples across the world; (b) Visualization of our S2 tokens.
• Figure 2: The GeoToken pipeline for retrieval-augmented geolocalization. A query image is encoded (1) and used to retrieve visually similar neighbors and their S2 location tokens from a gallery (2). This retrieved context grounds an encoder-decoder Transformer (3) that autoregressively predicts the final location as a hierarchical S2 token sequence. At test time, a pool of candidate locations is generated and a final prediction is chosen using a reranking strategy (4).
• Figure 3: Comparison of the median localization error (km) on YFCC4K and IM2GPS3K of GeoToken and prior state-of-the-art approaches. Left: MLLM-Free (GeoToken vs. PIGEON). Right: MLLM-Assisted (GeoToken vs. G3).
• Figure 4: Median error of best closest-in-pool using different numbers of samples and temperatures on both datasets.