SAMChat: Introducing Chain of Thought Reasoning and GRPO to a Multimodal Small Language Model for Small Scale Remote Sensing

TL;DR

This work tackles the challenge of deploying domain-adapted, resource-efficient multimodal reasoning for remote sensing in clandestine areas. It introduces SAMChat, a 2B parameter multimodal small language model, trained with supervised fine-tuning on expert-verified SAM imagery and enhanced by chain-of-thought reasoning and Group Relative Policy Optimization to improve detection of military installations. The authors present the SAMData dataset and demonstrate that SAMChat, particularly the SAMChat-R1 variant, achieves superior open-ended captioning and detection performance compared with larger generalist models, while remaining suitable for edge deployment. The results highlight the value of targeted fine-tuning and reinforcement learning in specialized RS tasks, while acknowledging remaining challenges from camouflage and civilian-lookalike structures.

Abstract

Remarkable capabilities in understanding and generating text-image content have been demonstrated by recent advancements in multimodal large language models (MLLMs). However, their effectiveness in specialized domains-particularly those requiring resource-efficient and domain-specific adaptations-has remained limited. In this work, a lightweight multimodal language model termed SAMChat is introduced, specifically adapted to analyze remote sensing imagery in secluded areas, including challenging missile launch sites. A new dataset, SAMData, was compiled by verifying hundreds of aerial images through expert review, and subtle military installations were highlighted via detailed captions. Supervised fine-tuning on a 2B parameter open-source MLLM with chain-of-thought (CoT) reasoning annotations was performed, enabling more accurate and interpretable explanations. Additionally, Group Relative Policy Optimization (GRPO) was leveraged to enhance the model's ability to detect critical domain-specific cues-such as defensive layouts and key military structures-while minimizing false positives on civilian scenes. Through empirical evaluations, it has been shown that SAMChat significantly outperforms both larger, general-purpose multimodal models and existing remote sensing adapted approaches on open-ended captioning and classification metrics. Over 80% recall and 98% precision were achieved on the newly proposed SAMData benchmark, underscoring the potency of targeted fine-tuning and reinforcement learning in specialized real-world applications.

Figures (4)

• Figure 1: Proposed SAMChat, a CoT and GRPO powered language model for remote sensing, provides significant improvements on secluded area captioning, specifically, on military areas and missile launch sites.
• Figure 2: Some diverse examples of aerial imagery of SAM sites.
• Figure 3: Training pipeline for the proposed SAMChat family. (a) Satellite images containing expert-verified surface-to-air missile (SAM) sites and randomly sampled residential areas are processed by existing multimodal LLMs (MLLMs) to obtain both brief descriptions and chain-of-thought (CoT) captions. Starting from Qwen2-VL-2B, we train either (b) via supervised fine-tuning (SFT) on brief captions to obtain the base SAMChat model, or (c) via zero-shot reinforcement learning using Group Relative Policy Optimization (GRPO) with only expert-verified yes/no labels (no captions) to obtain SAMChat-Zero. (d) The final model, SAMChat-R1, is produced by first applying SFT on CoT captions and then further aligning with GRPO.
• Figure 4: Qualitative comparison of base SAMChat, SAMChat with CoT fine-tuning and SAMChat-R1 responses for two examples of SAMData-300-Test dataset. Reasoning steps are shown above the final concise answer.