Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UrbanCross: Enhancing Satellite Image-Text Retrieval with Cross-Domain Adaptation

Siru Zhong, Xixuan Hao, Yibo Yan, Ying Zhang, Yangqiu Song, Yuxuan Liang

TL;DR

UrbanCross tackles the challenge of cross-domain satellite image-text retrieval by integrating geo-tags into text generation via Large Multimodal Models and applying Segment Anything Model-based segmentation to refine visual features. The framework combines image-text-segment alignment with adaptive domain adaptation, using a curriculum-based source sampler and weighted adversarial fine-tuning to bridge geographic distribution gaps. Key contributions include data augmentation with geo-aware captions, fine-grained segmentation-guided alignment, and a novel domain adaptation pipeline that significantly improves cross-country retrieval performance. The results demonstrate substantial gains in retrieval accuracy and domain adaptability across diverse urban environments, highlighting practical utility for global urban analytics and geo-aware information access.

Abstract

Urbanization challenges underscore the necessity for effective satellite image-text retrieval methods to swiftly access specific information enriched with geographic semantics for urban applications. However, existing methods often overlook significant domain gaps across diverse urban landscapes, primarily focusing on enhancing retrieval performance within single domains. To tackle this issue, we present UrbanCross, a new framework for cross-domain satellite image-text retrieval. UrbanCross leverages a high-quality, cross-domain dataset enriched with extensive geo-tags from three countries to highlight domain diversity. It employs the Large Multimodal Model (LMM) for textual refinement and the Segment Anything Model (SAM) for visual augmentation, achieving a fine-grained alignment of images, segments and texts, yielding a 10% improvement in retrieval performance. Additionally, UrbanCross incorporates an adaptive curriculum-based source sampler and a weighted adversarial cross-domain fine-tuning module, progressively enhancing adaptability across various domains. Extensive experiments confirm UrbanCross's superior efficiency in retrieval and adaptation to new urban environments, demonstrating an average performance increase of 15% over its version without domain adaptation mechanisms, effectively bridging the domain gap.

UrbanCross: Enhancing Satellite Image-Text Retrieval with Cross-Domain Adaptation

TL;DR

UrbanCross tackles the challenge of cross-domain satellite image-text retrieval by integrating geo-tags into text generation via Large Multimodal Models and applying Segment Anything Model-based segmentation to refine visual features. The framework combines image-text-segment alignment with adaptive domain adaptation, using a curriculum-based source sampler and weighted adversarial fine-tuning to bridge geographic distribution gaps. Key contributions include data augmentation with geo-aware captions, fine-grained segmentation-guided alignment, and a novel domain adaptation pipeline that significantly improves cross-country retrieval performance. The results demonstrate substantial gains in retrieval accuracy and domain adaptability across diverse urban environments, highlighting practical utility for global urban analytics and geo-aware information access.

Abstract

Urbanization challenges underscore the necessity for effective satellite image-text retrieval methods to swiftly access specific information enriched with geographic semantics for urban applications. However, existing methods often overlook significant domain gaps across diverse urban landscapes, primarily focusing on enhancing retrieval performance within single domains. To tackle this issue, we present UrbanCross, a new framework for cross-domain satellite image-text retrieval. UrbanCross leverages a high-quality, cross-domain dataset enriched with extensive geo-tags from three countries to highlight domain diversity. It employs the Large Multimodal Model (LMM) for textual refinement and the Segment Anything Model (SAM) for visual augmentation, achieving a fine-grained alignment of images, segments and texts, yielding a 10% improvement in retrieval performance. Additionally, UrbanCross incorporates an adaptive curriculum-based source sampler and a weighted adversarial cross-domain fine-tuning module, progressively enhancing adaptability across various domains. Extensive experiments confirm UrbanCross's superior efficiency in retrieval and adaptation to new urban environments, demonstrating an average performance increase of 15% over its version without domain adaptation mechanisms, effectively bridging the domain gap.
Paper Structure (40 sections, 9 equations, 11 figures, 3 tables)

This paper contains 40 sections, 9 equations, 11 figures, 3 tables.

Table of Contents

  1. Introduction
  2. PRELIMINARIES
  3. Formulation
  4. Related Work
  5. Satellite Image-Text Retrieval
  6. Large Multimodal Model and Segment Anything Model
  7. Domain adaptation in Urban Research
  8. METHODOLOGY
  9. Image Captioning and Segmentation
  10. Text Augmentation with LMM and Geo-Tags
  11. Image Augmentation with SAM
  12. Multi-modal Pre-training
  13. Learning Satellite Image Representations
  14. Learning Text Representations
  15. Pair-wise Modal Alignment
  16. ...and 25 more sections

Figures (11)

  • Figure 1: (a) Domain adaptation for differentiating solar power plants from residential houses across countries. (b) Geo-tags can complement visual intuition.
  • Figure 2: Overall framework of the proposed UrbanCross.
  • Figure 3: The similarity matrix between source/target image-text pairs gradually transforms into the weight vector.
  • Figure 4: UrbanCross dataset statistics.
  • Figure 5: Impact of hyperparameters on Mean Recall during UrbanCross Pre-training (a-d) and Fine-tuning (e-g).
  • ...and 6 more figures