Uni-RS: A Spatially Faithful Unified Understanding and Generation Model for Remote Sensing

TL;DR

This work tackles the Spatial Reversal Curse in unified remote sensing multimodal models, where spatial relations are reliably captured during understanding but poorly executed during text-to-image generation. It introduces Uni-RS, a unified RS Vision–Language model that couples a multimodal LLM with diffusion-based generation via 256 learnable queries and a Transformer connector, augmented by three spatial mechanisms: Spatial-Layout Planning, Spatial-Aware Query Supervision, and Image–Caption Spatial Layout Variation. A new RS-Spatial dataset provides structured spatial layouts, spatial relation labels, and rotation-consistent captions to support training. Empirical results on RSICD, RSIEval, and VRSBench show substantial gains in spatial faithfulness for generation while preserving competitive understanding performance, demonstrating the practicality and importance of explicitly modeling spatial layouts in RS multimodal systems.

Abstract

Unified remote sensing multimodal models exhibit a pronounced spatial reversal curse: Although they can accurately recognize and describe object locations in images, they often fail to faithfully execute the same spatial relations during text-to-image generation, where such relations constitute core semantic information in remote sensing. Motivated by this observation, we propose Uni-RS, the first unified multimodal model tailored for remote sensing, to explicitly address the spatial asymmetry between understanding and generation. Specifically, we first introduce explicit Spatial-Layout Planning to transform textual instructions into spatial layout plans, decoupling geometric planning from visual synthesis. We then impose Spatial-Aware Query Supervision to bias learnable queries toward spatial relations explicitly specified in the instruction. Finally, we develop Image-Caption Spatial Layout Variation to expose the model to systematic geometry-consistent spatial transformations. Extensive experiments across multiple benchmarks show that our approach substantially improves spatial faithfulness in text-to-image generation, while maintaining strong performance on multimodal understanding tasks like image captioning, visual grounding, and VQA tasks.

Figures (4)

• Figure 1: Illustration of the Spatial Reversal Curse in unified remote sensing multimodal models. While the model correctly describes spatial relations from an input image (left), it often fails to faithfully execute the same relations during text-to-image generation (right).
• Figure 2: Quantitative characterization of the Spatial Reversal Curse. The baseline MetaQuery* model exhibits a pronounced gap between spatial understanding and spatially faithful text-to-image generation. The * denotes that the model was trained by us under the same experimental settings as described in Section 5.1 due to the absence of public checkpoints. The Uni-RS lagrely reduces this gap after incorporating the proposed spatial mechanisms.
• Figure 3: Overview of the Uni-RS framework. Uni-RS integrates a multimodal large language model (MLLM) with a diffusion-based image generator through learnable queries and a Transformer-based connector. During training, the framework is further enhanced by Spatial-Layout Planning, Spatial-Aware Query Supervision, and geometry-consistent image–caption transformations, which jointly reinforce instruction-level spatial constraints and improve spatial faithfulness during generation.
• Figure 4: Qualitative comparison of spatial faithfulness in text-to-image generation. BLIP-3o is a general-domain unified multimodal model for understanding and generation, while CRS-Diff and Text2Earth are remote-sensing-specific text-to-image generation models. Text2Earth model is fine-tuned using RSIEval and VRSBench dataset to produce $512\times512$ images.