SPEX: A Vision-Language Model for Land Cover Extraction on Spectral Remote Sensing Images

TL;DR

SPEX is the first multimodal vision-language model dedicated to land cover extraction in spectral remote sensing imagery and is capable of generating textual explanations for its predictions, thereby enhancing interpretability and user-friendliness.

Abstract

Spectral information has long been recognized as a critical cue in remote sensing observations. Although numerous vision-language models have been developed for pixel-level interpretation, spectral information remains underutilized, resulting in suboptimal performance, particularly in multispectral scenarios. To address this limitation, we construct a vision-language instruction-following dataset named SPIE, which encodes spectral priors of land-cover objects into textual attributes recognizable by large language models (LLMs), based on classical spectral index computations. Leveraging this dataset, we propose SPEX, a multimodal LLM designed for instruction-driven land cover extraction. To this end, we introduce several carefully designed components and training strategies, including multiscale feature aggregation, token context condensation, and multispectral visual pre-training, to achieve precise and flexible pixel-level interpretation. To the best of our knowledge, SPEX is the first multimodal vision-language model dedicated to land cover extraction in spectral remote sensing imagery. Extensive experiments on five public multispectral datasets demonstrate that SPEX consistently outperforms existing state-of-the-art methods in extracting typical land cover categories such as vegetation, buildings, and water bodies. Moreover, SPEX is capable of generating textual explanations for its predictions, thereby enhancing interpretability and user-friendliness. Code will be released at: https://github.com/MiliLab/SPEX.

Figures (13)

• Figure 1: Comparison of various methods for land cover extraction. (a) combines traditional spectral index techniques with machine learning methods; (b) directly trains segmentation networks on multispectral images; (c) first pre-trains visual models on large-scale images, which are then used as backbones for segmentation networks; (d) further extends this approach by introducing a language model, enabling flexible and interactive interpretations guided by textual instructions.
• Figure 2: An examples for response generation using designed system prompts and auxiliary instructions.
• Figure 3: The pipeline for constructing SPIE, it consists of two key stages: response generation and spectral information integration.
• Figure 4: Overall workflow of the proposed method: (a) Visual pre-training; (b) Architecture of SPEX.
• Figure 5: The diagram of MSAM.