LaVIDE: A Language-Vision Discriminator for Detecting Changes in Satellite Image with Map References

TL;DR

LaVIDE tackles cross-modal change detection when only a single satellite image and a map reference are available. It aligns maps and images in a language-vision feature space by converting maps to textual labels with ensemble prompts and enriching them with object-context information, while the image branch is guided by a SegFormer backbone and knowledge distillation to the language-vision space. A Mixture-of-Experts discriminative module compares linguistic and visual embeddings across multiple semantic perspectives to produce a change map $\mathbf{B} \in \{0,1\}^{H\times W}$. On four benchmarks (DynamicEarthNet, HRSCD, BANDON, SECOND), LaVIDE achieves state-of-the-art results with notable IoU gains (e.g., $+13.8\%$ on DynamicEarthNet and $+4.3\%$ on SECOND), demonstrating robust cross-modal alignment and preservation of high-level map semantics for improved change detection.

Abstract

Change detection, which typically relies on the comparison of bi-temporal images, is significantly hindered when only a single image is available. Comparing a single image with an existing map, such as OpenStreetMap, which is continuously updated through crowd-sourcing, offers a viable solution to this challenge. Unlike images that carry low-level visual details of ground objects, maps convey high-level categorical information. This discrepancy in abstraction levels complicates the alignment and comparison of the two data types. In this paper, we propose a \textbf{La}nguage-\textbf{VI}sion \textbf{D}iscriminator for d\textbf{E}tecting changes in satellite image with map references, namely \ours{}, which leverages language to bridge the information gap between maps and images. Specifically, \ours{} formulates change detection as the problem of ``{\textit Does the pixel belong to [class]?}'', aligning maps and images within the feature space of the language-vision model to associate high-level map categories with low-level image details. Moreover, we build a mixture-of-experts discriminative module, which compares linguistic features from maps with visual features from images across various semantic perspectives, achieving comprehensive semantic comparison for change detection. Extensive evaluation on four benchmark datasets demonstrates that \ours{} can effectively detect changes in satellite image with map references, outperforming state-of-the-art change detection algorithms, e.g., with gains of about $13.8$\% on the DynamicEarthNet dataset and $4.3$\% on the SECOND dataset.

Figures (3)

• Figure 1: Motivation illustration. (a) Category discrimination detects changes via semantic segmentation and label comparison, i.e., "What is the semantic category of each pixel?". Its results rely heavily on classification accuracy. (b) Vision discrimination compares visual details between maps and images for change detection, simplifying the problem to "Are the two pixels visually similar?". It struggles to utilize map semantics and is impacted by inherent visual differences. (c) Our language-vision discrimination identifies changes by comparing linguistic information from maps with visual details from images, further simplifying to "Does this pixel belong to [class]?", achieving more convincing results.
• Figure 2: The pipeline of our proposed LaVIDE, which leverages language to bridge the information gap between maps and images. Specifically, LaVIDE is structured with two parallel branches that separately encode maps and images. In the map branch, we utilize language to indicate ground objects, converting a map into a textual representation $\mathbf{L}$, and subsequently extract text embedding $\mathbf{T}$ with the text encoder of the language-vision model. To enrich the categorical information of the map, we design an object context optimization strategy that refines textual embedding by incorporating object attributes. For the image branch, LaVIDE relates the hierarchical encoder with the feature space of language-vision models, $\mathbf{G}_{LVM}$, ensuring the semantic alignment of vision embedding with text embedding. After that, we compare the object-specific text embedding $\mathbf{G}_{t}$ with vision embedding $\mathbf{G}_{v}$ using the MoE discriminative module, comprehensively comparing their semantic differences from various perspectives for change detection.
• Figure 3: Visualization of change detection. We choose two typical scenes from the DynamicEarthNet dataset, the top-two rows and the bottom-two rows, for comparisons. For each scene, starting from the top-left to the bottom-right, the figures are respectively the pre-change map, post-change image, post-change map, ground-truth (GT), SETR_PUP setrpup2021, Segformer segformer2021, SSG2 ssg22024, SNUNet snunet2021, CGNet cgnet2023, FHD fhd2022, ChangerEx changer2023, ChangeFormer changeformer2022, Mapformer mapformer2023, and our LaVIDE. The red boxes mark representative regions.