Semantic-Spatial Feature Fusion with Dynamic Graph Refinement for Remote Sensing Image Captioning

TL;DR

This paper tackles remote sensing image captioning by bridging visual and textual modalities through semantic-spatial feature fusion and dynamic graph refinement. The SFDR framework combines SSFF, which fuses CLIP-derived semantic features with grid and ROI representations, and DGFR, which uses a graph attention network with a dynamic weighting scheme to highlight scene-relevant objects. The method yields a refined feature set and a knowledge-guided attention mechanism that improves caption quality, with strong gains in CIDEr and BLEU metrics across Sydney-Captions, UCM-Captions, and RSICD datasets. The approach demonstrates the value of integrating cross-modal semantics with multi-level RSI representations to produce more accurate, detailed, and contextually grounded captions, offering practical impact for RS analysis and multimodal documentation.

Abstract

Remote sensing image captioning aims to generate semantically accurate descriptions that are closely linked to the visual features of remote sensing images. Existing approaches typically emphasize fine-grained extraction of visual features and capturing global information. However, they often overlook the complementary role of textual information in enhancing visual semantics and face challenges in precisely locating objects that are most relevant to the image context. To address these challenges, this paper presents a semantic-spatial feature fusion with dynamic graph refinement (SFDR) method, which integrates the semantic-spatial feature fusion (SSFF) and dynamic graph feature refinement (DGFR) modules. The SSFF module utilizes a multi-level feature representation strategy by leveraging pre-trained CLIP features, grid features, and ROI features to integrate rich semantic and spatial information. In the DGFR module, a graph attention network captures the relationships between feature nodes, while a dynamic weighting mechanism prioritizes objects that are most relevant to the current scene and suppresses less significant ones. Therefore, the proposed SFDR method significantly enhances the quality of the generated descriptions. Experimental results on three benchmark datasets demonstrate the effectiveness of the proposed method. The source code will be available at https://github.com/zxk688}{https://github.com/zxk688.

Figures (8)

• Figure 1: The motivation of our research. (a) Traditional RSIC methods primarily rely on single-level visual features to generate captions. (b) In comparison, we use multi-level feature representations to provide a more comprehensive understanding of semantic information, spatial scenes, and object details. (c) The challenge lies in the interference from unrelated objects in a heterogeneous background, which affects the model’s ability to accurately describe the target regions.
• Figure 2: Overall structure of the proposed SFDR. (a) Semantic-spatial feature fusion module, (b) Dynamic graph feature refinement module.
• Figure 3: Illustration of the designed DGFR module. In this module, features are treated as nodes, and their relationships are modeled within a graph network. Through weight computation, threshold filtering, and the construction of a weight matrix, feature nodes are reselected. Finally, features are refined based on each node and its neighboring edges.
• Figure 4: Visualization of the proposed caption generator.
• Figure 5: Data samples in the (a) Sydney-Captions dataset, (b) UCM-Captions dataset, and (c) RSICD dataset.