MV-CC: Mask Enhanced Video Model for Remote Sensing Change Caption

TL;DR

This paper introduces a novel video model-based paradigm without design of the fusion module and proposes a Mask-enhanced Video model for Change Caption (MV-CC), which uses the off-the-shelf video encoder to simultaneously extract the temporal and spatial features of bi-temporal images.

Abstract

Remote sensing image change caption (RSICC) aims to provide natural language descriptions for bi-temporal remote sensing images. Since Change Caption (CC) task requires both spatial and temporal features, previous works follow an encoder-fusion-decoder architecture. They use an image encoder to extract spatial features and the fusion module to integrate spatial features and extract temporal features, which leads to increasingly complex manual design of the fusion module. In this paper, we introduce a novel video model-based paradigm without design of the fusion module and propose a Mask-enhanced Video model for Change Caption (MV-CC). Specifically, we use the off-the-shelf video encoder to simultaneously extract the temporal and spatial features of bi-temporal images. Furthermore, the types of changes in the CC are set based on specific task requirements, and to enable the model to better focus on the regions of interest, we employ masks obtained from the Change Detection (CD) method to explicitly guide the CC model. Experimental results demonstrate that our proposed method can obtain better performance compared with other state-of-the-art RSICC methods. The code is available at https://github.com/liuruixun/MV-CC.

Figures (4)

• Figure 1: (a) Classical three-stage architecture, including an image encoder, a feature fusion module, and a language decoder. (b) our proposed video-based two-stage architecture. (c) demonstrates the performance of different backbones when the fusion module is omitted.
• Figure 2: The results of zero-shot inference using InternVideo2. Bold text represents the content of interest, and red text represents the content that is not of interest.
• Figure 3: The detailed illustration of MV-CC. The pre-trained video encoder is applied with LoRA loralowrankadaptationlarge fine-tuning to extract temporal features, followed by a simple projector to generate corresponding tokens. The mask obtained from the CD model is downsampled and multiplied with the tokens to produce new tokens (tokens in gray represent those that have been deactivated.). These tokens are then fed into a language decoder.
• Figure 4: Change captioning examples generated by MV-CC, which demonstrate the difference in descriptions with and without mask guidance in the LEVIR-MCI dataset.