A Recipe for Improving Remote Sensing VLM Zero Shot Generalization
Aviad Barzilai, Yotam Gigi, Amr Helmy, Vered Silverman, Yehonathan Refael, Bolous Jaber, Tomer Shekel, George Leifman, Genady Beryozkin
TL;DR
The paper tackles the paucity of diverse, richly described remote-sensing data for vision-language models by introducing two RS-specific datasets, RS-WebLI and RS-Landmarks, and training an 800M-parameter MaMMUT-based VLM with a 400M vision encoder and 400M language model. It demonstrates state-of-the-art zero-shot cross-modal retrieval on public benchmarks and investigates unseen-class generalization via holdout experiments, using a curriculum that blends both RS datasets. Beyond retrieval, the work outlines a self-supervised localization approach that distills image-level contrastive knowledge into region-level segmentation through pseudo-labeling and a novel Smooth-Attention-Operation to stabilize attention maps. Collectively, these results advance open-vocabulary remote sensing tasks and suggest efficient deployment potential on resource-constrained devices, with ongoing work aimed at improving localization via iterative refinement.
Abstract
Foundation models have had a significant impact across various AI applications, enabling use cases that were previously impossible. Contrastive Visual Language Models (VLMs), in particular, have outperformed other techniques in many tasks. However, their prevalence in remote sensing (RS) is still limited, due to the scarcity of diverse remote-sensing visual-language datasets. In this work we introduce two novel image-caption datasets for training of remote sensing foundation models. The first dataset pairs aerial and satellite imagery with captions generated by Gemini using landmarks extracted from Google Maps. The second dataset utilizes public web images and their corresponding alt-text, filtered for the remote sensing domain, resulting in a diverse dataset with greater breadth in image styles and subject matter. These datasets are used to pre-train the MaMMUT~\citep{kuo2023mammutsimplearchitecturejoint} VLM architecture, resulting in state-of-the-art generalization performance in zero-shot cross-modal retrieval on well-known public benchmarks. Finally, we present our ongoing research to distill image-level knowledge gained in the VLM contrastive training procedure to enhance the model's localization ability. Specifically, we iteratively generate pseudo-labels for image regions based on the model's attention maps and use these labels for further training. To mitigate noisy attention maps and create robust segmentation masks, we introduce a novel attention-pooling mechanism called the Smooth-Attention-Operation.