GeoDiT: A Diffusion-based Vision-Language Model for Geospatial Understanding

TL;DR

Geospatial understanding with remote-sensing vision-language models is hampered by autoregressive generation. The authors introduce GeoDiT, a diffusion-based, non-autoregressive model that performs parallel iterative refinement for geospatial text conditioned on visual context. They propose a two-stage training regime (vision-language alignment and full instruction tuning) and a low-confidence remasking inference strategy. Experiments across captioning, visual grounding, object detection, and RSVQA demonstrate state-of-the-art performance on structured, object-centric outputs, validating the advantage of aligning the generative process with geospatial data structure.

Abstract

Autoregressive models are structurally misaligned with the inherently parallel nature of geospatial understanding, forcing a rigid sequential narrative onto scenes and fundamentally hindering the generation of structured and coherent outputs. We challenge this paradigm by reframing geospatial generation as a parallel refinement process, enabling a holistic, coarse-to-fine synthesis that resolves all semantic elements simultaneously. To operationalize this, we introduce GeoDiT, the first diffusion-based vision-language model tailored for the geospatial domain. Extensive experiments demonstrate that GeoDiT establishes a new state-of-the-art on benchmarks requiring structured, object-centric outputs. It achieves significant gains in image captioning, visual grounding, and multi-object detection, precisely the tasks where autoregressive models falter. Our work validates that aligning the generative process with the data's intrinsic structure is key to unlocking superior performance in complex geospatial analysis.

Figures (8)

• Figure 1: Conceptual comparison of text generation paradigms for geospatial understanding. (a) A two-tower model based on contrastive learning. (b) An autoregressive model generating text sequentially. (c) Our proposed diffusion-based model generating text in parallel via iterative mask prediction.
• Figure 2: Overview of the GeoDiT framework, illustrating the (a) training and (b) inference procedures. (a) During training, GeoDiT is optimized with a mask-and-predict objective. The model learns to reconstruct the original text from a randomly masked version, conditioned on a prompt and the visual features from a SigLIP-2 encoder. This follows a two-stage strategy: initial vision-language alignment by training only the MLP projector (Stage I), followed by end-to-end instruction tuning of the entire model (Stage II). (b) Inference is a non-autoregressive, iterative refinement process. Starting from a fully masked template, the model repeatedly predicts the full sequence and then applies a low-confidence remasking strategy. High-confidence tokens are preserved while uncertain ones are re-masked for the subsequent refinement step, a process that continues for a fixed number of iterations to produce the final output.
• Figure 3: Illustration of the core mask-and-predict mechanism. (a) Training phase: The model is optimized to reconstruct the original text Labels from a randomly masked version Inputs. The loss is calculated only on the tokens that were masked. (b) Test phase: Generation is a non-sequential, iterative process. It starts from a fully masked sequence ($t=2$) and progressively predicts the full sentence over a series of refinement steps ($t=1$, $t=0$).
• Figure 4: Qualitative examples of the generative failure mode in autoregressive models for object detection
• Figure 5: Visualization of the hierarchical generation process of GeoDiT. The color of each token corresponds to its relative finalization step during the iterative inference process: yellow indicates early-stage tokens, pink indicates middle-stage tokens, and blue indicates late-stage tokens.