Efficient Few-Shot Learning in Remote Sensing: Fusing Vision and Vision-Language Models

TL;DR

The paper tackles the challenge of data-scarce, context-rich remote-sensing interpretation by fusing YOLO-based object detection with Vision-Language Models (LLaVA, Gemini, ChatGPT). The proposed pipeline uses YOLO to ground VLM reasoning, enabling accurate aircraft counting and nuanced scene understanding under raw and degraded imagery in few-shot settings. Quantitative results show an average $ ext{MAE}$ improvement of $48.46\%$ and a CLIPScore gain of $6.17\%$, with Gemini offering particularly strong improvements in degraded conditions; qualitative gains include richer, context-aware descriptions. The work demonstrates a practical, flexible framework for efficient remote-sensing analysis that can extend to disaster response and other geospatial tasks, potentially reducing labelled-data demands.

Abstract

Remote sensing has become a vital tool across sectors such as urban planning, environmental monitoring, and disaster response. While the volume of data generated has increased significantly, traditional vision models are often constrained by the requirement for extensive domain-specific labelled data and their limited ability to understand the context within complex environments. Vision Language Models offer a complementary approach by integrating visual and textual data; however, their application to remote sensing remains underexplored, particularly given their generalist nature. This work investigates the combination of vision models and VLMs to enhance image analysis in remote sensing, with a focus on aircraft detection and scene understanding. The integration of YOLO with VLMs such as LLaVA, ChatGPT, and Gemini aims to achieve more accurate and contextually aware image interpretation. Performance is evaluated on both labelled and unlabelled remote sensing data, as well as degraded image scenarios which are crucial for remote sensing. The findings show an average MAE improvement of 48.46% across models in the accuracy of aircraft detection and counting, especially in challenging conditions, in both raw and degraded scenarios. A 6.17% improvement in CLIPScore for comprehensive understanding of remote sensing images is obtained. The proposed approach combining traditional vision models and VLMs paves the way for more advanced and efficient remote sensing image analysis, especially in few-shot learning scenarios.

Figures (7)

• Figure 1: Raw and Degraded Image comparison example
• Figure 2: Precision-Recall Graph
• Figure 3: Illustration of different descriptions provided by different VLMs (Example 1)
• Figure 4: MAE Improvement over VLMs
• Figure 5: CLIPScore Improvement