RSBuilding: Towards General Remote Sensing Image Building Extraction and Change Detection with Foundation Model

TL;DR

RSBuilding tackles the challenge of cross-scene, dual-task building understanding in remote sensing by unifying building extraction and building change detection under a foundation-model–driven framework. It employs a Siamese dual-temporal encoder (ViT or Swin), a multi-level feature enhancer, and a cross-attention decoder with task prompts to produce three masks via a shared backbone, supported by a federated training strategy to cope with partially labeled data. The approach achieves state-of-the-art or competitive performance on multiple datasets (WHU, INRIA, LEVIR-CD, S2Looking, BANDON) and demonstrates strong zero-shot generalization on unseen datasets, validating cross-scene universality and task complementarity. The practical impact lies in scalable, cross-domain building understanding that can inform urban planning and change analysis with reduced data labeling requirements.

Abstract

The intelligent interpretation of buildings plays a significant role in urban planning and management, macroeconomic analysis, population dynamics, etc. Remote sensing image building interpretation primarily encompasses building extraction and change detection. However, current methodologies often treat these two tasks as separate entities, thereby failing to leverage shared knowledge. Moreover, the complexity and diversity of remote sensing image scenes pose additional challenges, as most algorithms are designed to model individual small datasets, thus lacking cross-scene generalization. In this paper, we propose a comprehensive remote sensing image building understanding model, termed RSBuilding, developed from the perspective of the foundation model. RSBuilding is designed to enhance cross-scene generalization and task universality. Specifically, we extract image features based on the prior knowledge of the foundation model and devise a multi-level feature sampler to augment scale information. To unify task representation and integrate image spatiotemporal clues, we introduce a cross-attention decoder with task prompts. Addressing the current shortage of datasets that incorporate annotations for both tasks, we have developed a federated training strategy to facilitate smooth model convergence even when supervision for some tasks is missing, thereby bolstering the complementarity of different tasks. Our model was trained on a dataset comprising up to 245,000 images and validated on multiple building extraction and change detection datasets. The experimental results substantiate that RSBuilding can concurrently handle two structurally distinct tasks and exhibits robust zero-shot generalization capabilities.

Figures (4)

• Figure 1: The overview structure of the RSBuilding, a model for processing dual-temporal images to obtain individual building masks and building changes. The model primarily consists of four parts: an encoder for extracting dual-temporal robust features, an enhancer for enriching multi-scale information, a decoder for conducting spatiotemporal information interaction and querying corresponding semantic mask features based on task clues, and a segmentation head for generating the final segmentation results by leveraging Einstein summation.
• Figure 2: Visual comparisons of the proposed method with other state-of-the-art methods for building extraction. Red means false positives (FP), while Blue denotes false negatives (FN). Samples are all from the WHU and INRIA building test sets.
• Figure 3: Visual comparisons of the proposed method with other state-of-the-art methods for change detection. Red means false positives (FP), while Blue denotes false negatives (FN). Samples are all from the LEVIR-CD and S2Looking test sets. The final two columns depict building extraction results from the pre-temporal and post-temporal images.
• Figure 4: Visual comparisons of the proposed method with other state-of-the-art methods for building extraction and change detection. Red means false positives (FP), while Blue denotes false negatives (FN). Samples are all from the BANDON test sets.