LISAT: Language-Instructed Segmentation Assistant for Satellite Imagery

TL;DR

LISAt advances geospatial AI by marrying vision-language reasoning with pixel-precise segmentation for satellite imagery. It introduces two datasets, PreGRES for broad instruction-tuning and GRES for semisynthetic reasoning segmentation, and demonstrates how an embedding-as-mask approach can yield high-quality segmentation masks from complex, implicit queries. Through a Remote-CLIP/Vicuna backbone and a SAM/GeoSAM segmentation head, LISAt achieves substantial gains over existing geospatial foundation models (e.g., BLEU-4 improvements on description tasks) and open-domain models on reasoning segmentation (e.g., gIoU gains), while maintaining open-source availability. The work lays groundwork for scalable, domain-specific multimodal reasoning in remote sensing, with practical implications for disaster response, urban planning, and environmental monitoring, and points to future directions in data diversification and integration with additional segmentation backbones and data modalities.

Abstract

Segmentation models can recognize a pre-defined set of objects in images. However, models that can reason over complex user queries that implicitly refer to multiple objects of interest are still in their infancy. Recent advances in reasoning segmentation--generating segmentation masks from complex, implicit query text--demonstrate that vision-language models can operate across an open domain and produce reasonable outputs. However, our experiments show that such models struggle with complex remote-sensing imagery. In this work, we introduce LISAt, a vision-language model designed to describe complex remote-sensing scenes, answer questions about them, and segment objects of interest. We trained LISAt on a new curated geospatial reasoning-segmentation dataset, GRES, with 27,615 annotations over 9,205 images, and a multimodal pretraining dataset, PreGRES, containing over 1 million question-answer pairs. LISAt outperforms existing geospatial foundation models such as RS-GPT4V by over 10.04 % (BLEU-4) on remote-sensing description tasks, and surpasses state-of-the-art open-domain models on reasoning segmentation tasks by 143.36 % (gIoU). Our model, datasets, and code are available at https://lisat-bair.github.io/LISAt/

Figures (10)

• Figure 1: Existing models struggle to generate accurate segmentation masks for complex natural language queries in remote-sensing imagery. LISAt, our open-source, open-data, foundation model for geospatial reasoning segmentation trained on GRES, our new semi-synthetic dataset for remote-sensing reasoning segmentation, helps to bridge the gap between SOTA reasoning segmentation models and remote-sensing domains.
• Figure 2: To generate synthetic data, we start with a seed detection dataset (xView). We then filter detections for those that are both visually interesting and highly distinguishable (A). For those detection, we then generate a natural language description (B), and a pixel-wise segmentation mask (C). Finally, the natural language description is used to generate a localization query (D). Together, the segmentation mask and the query form a ground-truth pair for the LISAt reasoning segmentation fine-tuning.
• Figure 3: LISAt integrates a geospatial multimodal large language model (MLLM) with a segmentation decoder to enable reasoning-based segmentation. LISAt first pre-trains a Remote-CLIP-based MLLM on PreGRES before fine-tuning on GRES. We then expand the LMM vocabulary with a segmentation token (<SEG>), whose final-layer embedding is projected into the SAM segmentation query space and combined with image features to produce a segmentation mask.
• Figure 4: Scaling behavior of LISAt on the GRES dataset. While adding additional data is helpful, even with $7K$training images (the full GRES dataset), we observe the beginning of a plateau in performance, particularly on cIOU scores. This suggests that more data alone may not be helpful, and instead, we may need additional data variance outside the xView classes.
• Figure 2.5: Class Distribution of 33% Training Set