Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GeoLLaVA: Efficient Fine-Tuned Vision-Language Models for Temporal Change Detection in Remote Sensing

Hosam Elgendy, Ahmed Sharshar, Ahmed Aboeitta, Yasser Ashraf, Mohsen Guizani

TL;DR

This work tackles temporal change detection in remote sensing by coupling an annotated video-frame-pair dataset derived from fMoW with parameter-efficient fine-tuning of vision-language models. By applying LoRA and QLoRA alongside pruning to Video-LLaVA and LLaVA-NeXT-Video, the authors achieve strong descriptive accuracy of land-use transformations (e.g., a BERT score of $0.864$ and ROUGE-1 of $0.576$ on 100K samples) while improving computational efficiency for potential real-time deployment. The dataset-rich approach, combined with ablation studies on LoRA configurations and quantization strategies, demonstrates practical pathways to scale temporal VLM capabilities in remote sensing. The findings suggest that carefully balanced PEFT and compression techniques can deliver accurate, efficient temporal descriptions suitable for environmental monitoring and urban planning tasks.

Abstract

Detecting temporal changes in geographical landscapes is critical for applications like environmental monitoring and urban planning. While remote sensing data is abundant, existing vision-language models (VLMs) often fail to capture temporal dynamics effectively. This paper addresses these limitations by introducing an annotated dataset of video frame pairs to track evolving geographical patterns over time. Using fine-tuning techniques like Low-Rank Adaptation (LoRA), quantized LoRA (QLoRA), and model pruning on models such as Video-LLaVA and LLaVA-NeXT-Video, we significantly enhance VLM performance in processing remote sensing temporal changes. Results show significant improvements, with the best performance achieving a BERT score of 0.864 and ROUGE-1 score of 0.576, demonstrating superior accuracy in describing land-use transformations.

GeoLLaVA: Efficient Fine-Tuned Vision-Language Models for Temporal Change Detection in Remote Sensing

TL;DR

This work tackles temporal change detection in remote sensing by coupling an annotated video-frame-pair dataset derived from fMoW with parameter-efficient fine-tuning of vision-language models. By applying LoRA and QLoRA alongside pruning to Video-LLaVA and LLaVA-NeXT-Video, the authors achieve strong descriptive accuracy of land-use transformations (e.g., a BERT score of and ROUGE-1 of on 100K samples) while improving computational efficiency for potential real-time deployment. The dataset-rich approach, combined with ablation studies on LoRA configurations and quantization strategies, demonstrates practical pathways to scale temporal VLM capabilities in remote sensing. The findings suggest that carefully balanced PEFT and compression techniques can deliver accurate, efficient temporal descriptions suitable for environmental monitoring and urban planning tasks.

Abstract

Detecting temporal changes in geographical landscapes is critical for applications like environmental monitoring and urban planning. While remote sensing data is abundant, existing vision-language models (VLMs) often fail to capture temporal dynamics effectively. This paper addresses these limitations by introducing an annotated dataset of video frame pairs to track evolving geographical patterns over time. Using fine-tuning techniques like Low-Rank Adaptation (LoRA), quantized LoRA (QLoRA), and model pruning on models such as Video-LLaVA and LLaVA-NeXT-Video, we significantly enhance VLM performance in processing remote sensing temporal changes. Results show significant improvements, with the best performance achieving a BERT score of 0.864 and ROUGE-1 score of 0.576, demonstrating superior accuracy in describing land-use transformations.

Paper Structure

This paper contains 15 sections, 8 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Introduction and Processing
  4. fMoW Dataset
  5. Creating Image Pairs
  6. Data Splits
  7. Temporal Annotations
  8. Experimental Setup
  9. Model Fine-tuning
  10. Low Rank Adaptation
  11. Evaluation Metrics
  12. Model Optimization
  13. Results & Discussion
  14. Conclusion
  15. Limitation & Future Work

Figures (5)

  • Figure 1: Overview of the video creation process from the original fMoW dataset images.
  • Figure 2: Overall sample and per-question distributions.
  • Figure 3: Word clouds comparing the ground truth (left) and the final model's generated annotations (right).
  • Figure 4: Qualitative output comparing the output from ChatGPT vs our model's for two images that look similar but for different locations, showcasing the ability of distinguishing differences
  • Figure 5: Qualitative output showcasing a sample video of two frames inputted with a question followed with the model output describing the two images and summarizing differences and changes.