Enhancing Perception of Key Changes in Remote Sensing Image Change Captioning

TL;DR

This work tackles RSICC by addressing the problem of irrelevant change features steering descriptions astray. It introduces KCFI, a multimodal framework that leverages a ViT-based feature extractor, a key-change feature perceiver, a pixel-level change detector, and an instruction-tuned large language model guided by visual instructions. A dynamic weight averaging scheme balances captioning and change detection losses, while experiments on LEVIR-CC demonstrate state-of-the-art performance across multiple evaluation metrics. The approach highlights the value of focusing on key change features and integrating visual instructions with LLMs to produce accurate, context-rich change descriptions suitable for practical remote sensing applications; code will be released at the provided repository.

Abstract

Recently, while significant progress has been made in remote sensing image change captioning, existing methods fail to filter out areas unrelated to actual changes, making models susceptible to irrelevant features. In this article, we propose a novel multimodal framework for remote sensing image change captioning, guided by Key Change Features and Instruction-tuned (KCFI). This framework aims to fully leverage the intrinsic knowledge of large language models through visual instructions and enhance the effectiveness and accuracy of change features using pixel-level change detection tasks. Specifically, KCFI includes a ViTs encoder for extracting bi-temporal remote sensing image features, a key feature perceiver for identifying critical change areas, a pixel-level change detection decoder to constrain key change features, and an instruction-tuned decoder based on a large language model. Moreover, to ensure that change description and change detection tasks are jointly optimized, we employ a dynamic weight-averaging strategy to balance the losses between the two tasks. We also explore various feature combinations for visual fine-tuning instructions and demonstrate that using only key change features to guide the large language model is the optimal choice. To validate the effectiveness of our approach, we compare it against several state-of-the-art change captioning methods on the LEVIR-CC dataset, achieving the best performance. Our code will be available at https://github.com/yangcong356/KCFI.git.

Figures (7)

• Figure 1: Visualization of irrelevant areas in the change features. (a) Pre-event remote sensing image; (b) Post-event remote sensing image; (c) Visualization of RSICCFormer’s LZC2022 change features; (d) Visualization of KCFI’s change features. The results in Fig. \ref{['ch4-fig3']} show that using only change features and textual instructions to guide the large language model in generating change descriptions is the optimal approach. Therefore, filtering out irrelevant change features is crucial for improving the model’s accuracy in change descriptions.
• Figure 2: The KCFI framework begins by extracting multi-level features from bi-temporal remote sensing images using a Vision Encoder. By processing these features, the key change perception module can identify key changes. The key change features replace the “< image>” token in the Change Caption Branch, where they are used to guide the large language model in generating change descriptions. Simultaneously, the Change Detection Branch processes the change features to generate change masks. A dynamic weight average balances the losses from both the captioning and change detection tasks to ensure optimal performance. The “flames” represent fine-tuning the network parameters, while the “ice cubes” represent freezing the network parameters.
• Figure 3: The flowchart of the change feature extractor illustrates the processing flow of multi-level features. First, the features extracted from the four stages are reshaped. Next, these reshaped features are passed in parallel through the key change feature extractor to obtain the final key change features.
• Figure 4: The schematic diagram of the change description branch. First, the instructions are encoded into word vectors, then the “< image>” token is replaced with the key change features. The modified features are input into the large language model to generate the change description.
• Figure 5: Change captioning results generated by RSICCFormer, PromptCC, Chg2Cap, and the KCFI on the LEVIR-CC dataset. GT represents one of the five ground truth annotations from the original dataset. Red ones indicate accurate descriptions, while those in blue represent incorrect descriptions.