GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensing

TL;DR

GRASP addresses the domain gap faced when adapting Multimodal Large Language Models to remote sensing imagery by introducing region-aware, sparse soft prompting that associates learnable prompts with spatial blocks in a frozen backbone. A question-guided sparse fusion mechanism aggregates block-level cues into a compact global prompt, injected as an additional visual token to steer the frozen language model without changing backbone parameters. The approach achieves competitive RSVQA/RSIVQA performance with substantial parameter efficiency, outperforming static prompts and showings strong region-specific reasoning across resolutions. This method offers a practical path for efficient RS adaptation of LLMs, enabling scalable deployment in RS tasks and potentially extending to temporal or multi-view data.

Abstract

In recent years, Multimodal Large Language Models (MLLMs) have made significant progress in visual question answering tasks. However, directly applying existing fine-tuning methods to remote sensing (RS) images often leads to issues such as overfitting on background noise or neglecting target details. This is primarily due to the large-scale variations, sparse target distributions, and complex regional semantic features inherent in RS images. These challenges limit the effectiveness of MLLMs in RS tasks. To address these challenges, we propose a parameter-efficient fine-tuning (PEFT) strategy called Guided Region-Aware Sparse Prompting (GRASP). GRASP introduces spatially structured soft prompts associated with spatial blocks extracted from a frozen visual token grid. Through a question-guided sparse fusion mechanism, GRASP dynamically aggregates task-specific context into a compact global prompt, enabling the model to focus on relevant regions while filtering out background noise. Extensive experiments on multiple RSVQA benchmarks show that GRASP achieves competitive performance compared to existing fine-tuning and prompt-based methods while maintaining high parameter efficiency.

Figures (6)

• Figure 1: Parameter efficiency vs. performance on Qwen2.5-VL-7B. The y-axis denotes the overall accuracy (mean of the AA scores on RSVQA-LR, RSVQA-HR, and RSIVQA), plotted against trainable parameters (log scale). Our method, GRASP, is represented by a red star. Note that proximity to the top-left corner corresponds to higher accuracy with lower parameter costs.
• Figure 2: Conceptual comparison between Conventional soft prompting and GRASP. (a) Conventional soft prompting: Relies on static, uniform prompts. It struggles to suppress background interference (e.g., $E_1$), as it lacks region-aware adaptation. (b) GRASP: Utilizes question-guided sparse weighting to suppress irrelevant backgrounds ($E_1$ weight=0) and emphasize targets ($E_4$ weight=0.5). The resulting global prompt $P_{global}$ is computed by weighted fusion over all regions, where regions with zero weights are ignored.
• Figure 3: Architecture of the GRASP framework. (a) Spatial Block-wise Soft Prompt Construction: The frozen vision encoder generates a visual token grid from the image. Average pooling partitions this grid into spatial blocks (e.g., $E_1$--$E_4$), where each block $E_i$ links to a learnable local soft prompt (SP). (b) Question-guided Sparse Soft Prompt Fusion: The frozen LLM derives a semantic vector $q$ from the question. The system projects $q$ and block-level features $E_i$ into a shared space to compute relevance scores. Entmax activation converts these scores into sparse weights $\{w_i\}$, which the model uses to aggregate local SPs into a task-specific global soft prompt $P_{global}$. (c) LLM-based Inference and Gradient Backpropagation: The framework injects $P_{global}$ into the multimodal token sequence to guide the frozen LLM in answer generation. Training updates only the soft prompts and linear projection parameters.
• Figure 4: Parameter efficiency vs. performance on LLaVA-Next. The y-axis denotes the overall accuracy (mean of the AA scores on RSVQA-LR, RSVQA-HR, and RSIVQA), plotted against trainable parameters (log scale). Our method, GRASP, is represented by a red star. Note that proximity to the top-left corner corresponds to higher accuracy with lower parameter costs.
• Figure 5: Visualization of Question-guided Sparse Fusion Weights across different input resolutions. The top row shows the $4 \times 4$ grid weights for the $512 \times 512$ input, while the bottom row displays the $2 \times 2$ grid for the $224 \times 224$ input. These weights represent the contribution of each spatial block to the final representation, with answers highlighted in green denoting correct predictions.