Multimodal Interpretation of Remote Sensing Images: Dynamic Resolution Input Strategy and Multi-scale Vision-Language Alignment Mechanism

TL;DR

This work tackles the efficiency-accuracy bottleneck in multimodal remote sensing interpretation by introducing a Dynamic Resolution Input Strategy (DRIS) and a Multi-scale Vision-language Alignment Mechanism (MS-VLAM) within a Vision-Language Model. DRIS enables coarse-to-fine processing to allocate compute where it matters, while MS-VLAM enforces hierarchical cross-modal alignment across object-level, local-region-level, and global-level representations, guided by a multi-term loss $\mathcal{L} = \mathcal{L}_{\text{caption}} + \delta \cdot \mathcal{L}_{\text{align}}$ and task-driven optimization. The framework achieves state-of-the-art results on RS-GPT4V and RS-VLM benchmarks, including improvements in BLEU-4, CIDEr, and R@10 metrics for captioning and retrieval, and higher Accuracy@0.5 for visual grounding, demonstrating robust cross-modal reasoning in diverse RS scenarios. This work advances practical RS interpretation by delivering an efficient, scalable, and semantically rich multimodal framework suitable for environmental monitoring, urban management, and disaster response.

Abstract

Multimodal fusion of remote sensing images serves as a core technology for overcoming the limitations of single-source data and improving the accuracy of surface information extraction, which exhibits significant application value in fields such as environmental monitoring and urban planning. To address the deficiencies of existing methods, including the failure of fixed resolutions to balance efficiency and detail, as well as the lack of semantic hierarchy in single-scale alignment, this study proposes a Vision-language Model (VLM) framework integrated with two key innovations: the Dynamic Resolution Input Strategy (DRIS) and the Multi-scale Vision-language Alignment Mechanism (MS-VLAM).Specifically, the DRIS adopts a coarse-to-fine approach to adaptively allocate computational resources according to the complexity of image content, thereby preserving key fine-grained features while reducing redundant computational overhead. The MS-VLAM constructs a three-tier alignment mechanism covering object, local-region and global levels, which systematically captures cross-modal semantic consistency and alleviates issues of semantic misalignment and granularity imbalance.Experimental results on the RS-GPT4V dataset demonstrate that the proposed framework significantly improves the accuracy of semantic understanding and computational efficiency in tasks including image captioning and cross-modal retrieval. Compared with conventional methods, it achieves superior performance in evaluation metrics such as BLEU-4 and CIDEr for image captioning, as well as R@10 for cross-modal retrieval. This technical framework provides a novel approach for constructing efficient and robust multimodal remote sensing systems, laying a theoretical foundation and offering technical guidance for the engineering application of intelligent remote sensing interpretation.

Figures (4)

• Figure 1: Overview of the proposed multimodal intelligent fusion framework for remote sensing applications. The framework first processes multi-source remote sensing data (optical imagery, SAR, LiDAR, hyperspectral) via the Dynamic Resolution Input Strategy (DRIS), which balances feature extraction accuracy and computational efficiency. Cross-modal semantic matching is then implemented through the Multi-scale Vision–language Alignment Mechanism (MS-VLAM), which decomposes alignment into Object-level, Local-region-level, and Global-level granularities to strengthen visual-textual consistency. This framework supports a variety of downstream tasks, including land cover classification, disaster response, and urban management.
• Figure 2: Workflow of the dynamic resolution visual-language fusion framework for remote sensing image captioning.This framework integrates a coarse-to-fine dynamic resolution input strategy (balancing computational efficiency and detail preservation), a multi-scale visual-text alignment module (matching object,local-region,global visual features with textual units), and a hierarchical fusion module. The fused visual-linguistic features are fed into a large language model (LLM) to generate semantically consistent descriptive captions (e.g., “Two large commercial airliners were parked on the apron”).
• Figure 3: The workflow of the proposed high-resolution remote sensing fine-grained processing framework. Starting from large-format remote sensing image input (after preprocessing), the pipeline first extracts low-resolution features and saliency maps, then selects regions of interest (ROIs) via threshold comparison and attention-based saliency mapping. High-resolution processing refines ROI blocks, followed by feature fusion using a ResNet-Transformer-FPN architecture. Finally, vision-language retrieval is implemented with BERT text embedding and ResNet image embedding, optimized by cross entropy and infoNCE losses.The bottom illustration visualizes the stepwise refinement from global scene understanding to local fine-grained analysis (e.g., small object/boundary detail extraction).
• Figure 4: The framework of Multi-scale Vision-language Alignment Mechanism (MS-VLAM). This framework comprises three core modules. First, the Visual Feature Extraction Module extracts visual features at three scales: object-level features via detector and ROI pooling, local-region-level features via SAM-based segmentation and masked pooling, and global-level features via spatial pyramid pooling. Second, the Text Feature Extraction Module generates corresponding text features from the text input, including object text features, phrase text features, and global CLS vector features. Third, the Loss Alignment Module conducts scale-specific vision-language alignment and calculates alignment losses for each scale. Finally, the weighted combinational loss (fused with the captioning loss) is applied to optimize the model, supporting downstream tasks such as image captioning, visual retrieval, and visual QA.