Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vision-Language Models in Remote Sensing: Current Progress and Future Trends

Xiang Li, Congcong Wen, Yuan Hu, Zhenghang Yuan, Xiao Xiang Zhu

TL;DR

This survey examines the rise of vision-language models in remote sensing, highlighting how combining visual and textual modalities enables semantic reasoning beyond traditional visual-only RS approaches. It categorizes RS VLM architectures into fusion-encoder and dual-encoder frameworks, surveys foundation-model efforts, and reviews RS-specific tasks including captioning, text-based generation and retrieval, VQA, grounding, and few-/zero-shot classification and segmentation. The authors catalog datasets, benchmarks, and open-source resources, and discuss current limitations such as dataset scale and domain gaps, proposing directions like unified RS VLMs, data-efficient fine-tuning, diffusion-based generation, and integration of RS expert knowledge. The work underscores the potential impact on practical RS applications, including change detection, disaster response, forest monitoring, and climate-related analyses, by enabling richer, language-assisted interpretation of Earth-observation data.

Abstract

The remarkable achievements of ChatGPT and GPT-4 have sparked a wave of interest and research in the field of large language models for Artificial General Intelligence (AGI). These models provide intelligent solutions close to human thinking, enabling us to use general artificial intelligence to solve problems in various applications. However, in remote sensing (RS), the scientific literature on the implementation of AGI remains relatively scant. Existing AI-related research in remote sensing primarily focuses on visual understanding tasks while neglecting the semantic understanding of the objects and their relationships. This is where vision-language models excel, as they enable reasoning about images and their associated textual descriptions, allowing for a deeper understanding of the underlying semantics. Vision-language models can go beyond visual recognition of RS images, model semantic relationships, and generate natural language descriptions of the image. This makes them better suited for tasks requiring visual and textual understanding, such as image captioning, and visual question answering. This paper provides a comprehensive review of the research on vision-language models in remote sensing, summarizing the latest progress, highlighting challenges, and identifying potential research opportunities.

Vision-Language Models in Remote Sensing: Current Progress and Future Trends

TL;DR

This survey examines the rise of vision-language models in remote sensing, highlighting how combining visual and textual modalities enables semantic reasoning beyond traditional visual-only RS approaches. It categorizes RS VLM architectures into fusion-encoder and dual-encoder frameworks, surveys foundation-model efforts, and reviews RS-specific tasks including captioning, text-based generation and retrieval, VQA, grounding, and few-/zero-shot classification and segmentation. The authors catalog datasets, benchmarks, and open-source resources, and discuss current limitations such as dataset scale and domain gaps, proposing directions like unified RS VLMs, data-efficient fine-tuning, diffusion-based generation, and integration of RS expert knowledge. The work underscores the potential impact on practical RS applications, including change detection, disaster response, forest monitoring, and climate-related analyses, by enabling richer, language-assisted interpretation of Earth-observation data.

Abstract

The remarkable achievements of ChatGPT and GPT-4 have sparked a wave of interest and research in the field of large language models for Artificial General Intelligence (AGI). These models provide intelligent solutions close to human thinking, enabling us to use general artificial intelligence to solve problems in various applications. However, in remote sensing (RS), the scientific literature on the implementation of AGI remains relatively scant. Existing AI-related research in remote sensing primarily focuses on visual understanding tasks while neglecting the semantic understanding of the objects and their relationships. This is where vision-language models excel, as they enable reasoning about images and their associated textual descriptions, allowing for a deeper understanding of the underlying semantics. Vision-language models can go beyond visual recognition of RS images, model semantic relationships, and generate natural language descriptions of the image. This makes them better suited for tasks requiring visual and textual understanding, such as image captioning, and visual question answering. This paper provides a comprehensive review of the research on vision-language models in remote sensing, summarizing the latest progress, highlighting challenges, and identifying potential research opportunities.
Paper Structure (22 sections, 4 equations, 16 figures, 11 tables)

This paper contains 22 sections, 4 equations, 16 figures, 11 tables.

Table of Contents

  1. Introduction
  2. From Vision-Centric to Vision-Language Models
  3. Vision-Centric Models
  4. Large Language Models
  5. Vision-Language Models
  6. Fusion Encoder
  7. Dual Encoder
  8. Vision-Language Models in Remote Sensing
  9. Vision-Centric Foundation Models
  10. Image Captioning
  11. Text-based Image Generation
  12. Text-based Image Retrieval
  13. Visual Question Answering
  14. Visual Grounding
  15. Zero-Shot Scene Classification.
  16. ...and 7 more sections

Figures (16)

  • Figure 1: Vision language models in remote sensing.
  • Figure 2: Architecture overview of ViT model vit
  • Figure 3: Network architecture and training objectives of the GPT model radford2018improving.
  • Figure 4: Pre-training objectives of the BERT model liu2019roberta.
  • Figure 5: Illustration of the architecture of VisualBERT li2019visualbert
  • ...and 11 more figures