Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GeoChat: Grounded Large Vision-Language Model for Remote Sensing

Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, Fahad Shahbaz Khan

TL;DR

GeoChat tackles the gap between general vision-language models and remote sensing data by introducing a grounded, multitask RS VLM. It leverages a CLIP-ViT backbone, Vicuna-LLM, and LoRA fine-tuning to support image- and region-level dialogue with explicit spatial grounding, powered by a large RS multimodal instruction dataset of ~318k pairs. The model demonstrates robust zero-shot performance across scene classification, VQA, and grounded conversations, and it provides a comprehensive RS multitask benchmark for future work. This work significantly advances versatile, conversation-capable RS AI with practical implications for remote-sensing analysis and decision support.

Abstract

Recent advancements in Large Vision-Language Models (VLMs) have shown great promise in natural image domains, allowing users to hold a dialogue about given visual content. However, such general-domain VLMs perform poorly for Remote Sensing (RS) scenarios, leading to inaccurate or fabricated information when presented with RS domain-specific queries. Such a behavior emerges due to the unique challenges introduced by RS imagery. For example, to handle high-resolution RS imagery with diverse scale changes across categories and many small objects, region-level reasoning is necessary alongside holistic scene interpretation. Furthermore, the lack of domain-specific multimodal instruction following data as well as strong backbone models for RS make it hard for the models to align their behavior with user queries. To address these limitations, we propose GeoChat - the first versatile remote sensing VLM that offers multitask conversational capabilities with high-resolution RS images. Specifically, GeoChat can not only answer image-level queries but also accepts region inputs to hold region-specific dialogue. Furthermore, it can visually ground objects in its responses by referring to their spatial coordinates. To address the lack of domain-specific datasets, we generate a novel RS multimodal instruction-following dataset by extending image-text pairs from existing diverse RS datasets. We establish a comprehensive benchmark for RS multitask conversations and compare with a number of baseline methods. GeoChat demonstrates robust zero-shot performance on various RS tasks, e.g., image and region captioning, visual question answering, scene classification, visually grounded conversations and referring detection. Our code is available at https://github.com/mbzuai-oryx/geochat.

GeoChat: Grounded Large Vision-Language Model for Remote Sensing

TL;DR

GeoChat tackles the gap between general vision-language models and remote sensing data by introducing a grounded, multitask RS VLM. It leverages a CLIP-ViT backbone, Vicuna-LLM, and LoRA fine-tuning to support image- and region-level dialogue with explicit spatial grounding, powered by a large RS multimodal instruction dataset of ~318k pairs. The model demonstrates robust zero-shot performance across scene classification, VQA, and grounded conversations, and it provides a comprehensive RS multitask benchmark for future work. This work significantly advances versatile, conversation-capable RS AI with practical implications for remote-sensing analysis and decision support.

Abstract

Recent advancements in Large Vision-Language Models (VLMs) have shown great promise in natural image domains, allowing users to hold a dialogue about given visual content. However, such general-domain VLMs perform poorly for Remote Sensing (RS) scenarios, leading to inaccurate or fabricated information when presented with RS domain-specific queries. Such a behavior emerges due to the unique challenges introduced by RS imagery. For example, to handle high-resolution RS imagery with diverse scale changes across categories and many small objects, region-level reasoning is necessary alongside holistic scene interpretation. Furthermore, the lack of domain-specific multimodal instruction following data as well as strong backbone models for RS make it hard for the models to align their behavior with user queries. To address these limitations, we propose GeoChat - the first versatile remote sensing VLM that offers multitask conversational capabilities with high-resolution RS images. Specifically, GeoChat can not only answer image-level queries but also accepts region inputs to hold region-specific dialogue. Furthermore, it can visually ground objects in its responses by referring to their spatial coordinates. To address the lack of domain-specific datasets, we generate a novel RS multimodal instruction-following dataset by extending image-text pairs from existing diverse RS datasets. We establish a comprehensive benchmark for RS multitask conversations and compare with a number of baseline methods. GeoChat demonstrates robust zero-shot performance on various RS tasks, e.g., image and region captioning, visual question answering, scene classification, visually grounded conversations and referring detection. Our code is available at https://github.com/mbzuai-oryx/geochat.
Paper Structure (12 sections, 2 equations, 5 figures, 10 tables)

This paper contains 12 sections, 2 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. GeoChat: Grounded Remote Sensing VLM
  4. GeoChat Architecture
  5. Training Details
  6. RS Multimodal Instruction Dataset
  7. Experiments
  8. Implementation Details
  9. Scene Classification
  10. Visual Question Answering
  11. Visual Grounding
  12. Conclusion

Figures (5)

  • Figure 1: GeoChat can accomplish multiple tasks for remote-sensing (RS) image comprehension in a unified framework. Given suitable task tokens and user queries, the model can generate visually grounded responses (text with corresponding object locations - shown on top), visual question answering on images and regions (top left and bottom right, respectively) as well as scene classification (top right) and normal natural language conversations (bottom). This makes it the first RS VLM with grounding capability.
  • Figure 2: An overview of GeoChat - the first grounded large vision-language model for remote sensing. Given an image input together with a user query, a visual backbone is first used to encode patch-level tokens at a higher resolution via interpolating positional encodings. A multi-layer perceptron (MLP) is used to adapt vision-tokens to language space suitable for input to a Large Language Model (Vicuna 1.5). Besides visual inputs, region locations can also be input to the model together with task-specific prompts that specify the desired task required by the user. Given this context, the LLM can generate natural language responses interleaved with corresponding object locations. GeoChat can perform multiple tasks as shown on top e.g., scene classification, image/region captioning, VQA and grounded conversations.
  • Figure 3: Multi-task instruction template for GeoChat.
  • Figure 4: Types of annotations available in the GeoChat instruction-set. Top-row: For a given RS image, we obtain object attribute and relationship information, referring expressions and region captions along with their corresponding region annotations (shown over the image). Bottom-row: This structured information is used to create the rich instruction-set with a total of 318k image-instruction pairs.
  • Figure 5: Qualitative results of GeoChat. (left-right) Results are shown on grounding, referring object detection, and disaster/damage detection. The user can provide task-specific tokens (e.g., [grounding]) to shape model responses according to the desired behavior. The model can generate textual responses (right), only visual grounding (center) and both text and object groundings interleaved together (left). The model can also specify object types, object counts, object attributes and object relationships.