EcoWikiRS: Learning Ecological Representation of Satellite Images from Weak Supervision with Species Observations and Wikipedia

TL;DR

EcoWikiRS addresses the challenge of learning ecologically meaningful representations from remote sensing images using weak supervision from species observations and Wikipedia habitat text. It introduces the EcoWikiRS dataset and the WINCEL loss to up-weight informative text while down-weighting noisy, irrelevant passages, enabling cross-modal alignment between very-high-resolution RS imagery and ecological descriptions. In zero-shot ecosystem mapping on the EUNIS habitat framework, WINCEL improves over standard InfoNCE baselines across multiple RS-VLM backbones, with particularly strong gains for SkyCLIP and CLIP, and demonstrates the ability to retrieve ecologically relevant sentences and generate coherent habitat maps. The work highlights practical considerations for dataset construction, text selection, and fine-tuning strategies, and points toward integrating ecological knowledge into RS-VLMs for more interpretable and actionable earth observation analyses.

Abstract

The presence of species provides key insights into the ecological properties of a location such as land cover, climatic conditions or even soil properties. We propose a method to predict such ecological properties directly from remote sensing (RS) images by aligning them with species habitat descriptions. We introduce the EcoWikiRS dataset, consisting of high-resolution aerial images, the corresponding geolocated species observations, and, for each species, the textual descriptions of their habitat from Wikipedia. EcoWikiRS offers a scalable way of supervision for RS vision language models (RS-VLMs) for ecology. This is a setting with weak and noisy supervision, where, for instance, some text may describe properties that are specific only to part of the species' niche or is irrelevant to a specific image. We tackle this by proposing WINCEL, a weighted version of the InfoNCE loss. We evaluate our model on the task of ecosystem zero-shot classification by following the habitat definitions from the European Nature Information System (EUNIS). Our results show that our approach helps in understanding RS images in a more ecologically meaningful manner. The code and the dataset are available at https://github.com/eceo-epfl/EcoWikiRS.

Figures (9)

• Figure 1: EcoWikiRS connects aerial images with local species observations from crowd-sourcing platforms. For each species, a habitat description from Wikipedia is retrieved and used for feature alignment. Through the proposed WINCEL loss, we learn to recognize text passages that are relevant to the images and integrating some ecological knowledge in the aerial images representation.
• Figure 2: EcoWikiRS dataset preparation. For each location containing species observation in GBIF, an aerial image and its ecosystem type from the EUNIS map are extracted. The corresponding Wikipedia articles are collected. The retrieved text is parsed and split into sentences before a filtering step aiming at keeping only ecosystem-related sentences.
• Figure 3: Visualization of text-image similarity values over the surface of Switzerland with different text prompts as inputs. The maps uses pretrained SkyCLIP at the top and fine-tuned SkyCLIP at the bottom. The simplified land cover map and the temperature maps are for reference. High similarity values are shown in green, while magenta depicts lower values with min-max scaling. Best viewed in colors.
• Figure 4: Comparison of top-3 sentences scores given by the pretrained and fine-tuned SkyCLIP model on samples of the WikiRS dataset. Scores are cosine similarity values between the text and images features. Scores are ranked by decreasing order of magnitude. More examples are available in the Suppl. Figure \ref{['fig:simi_suppl']}.
• Figure 5: Cross-modal similarities values between sentences from the Wikipedia article of Turdus merula (common black bird) and various images from our dataset representing different land cover. High similarity values are shown in green, and low similarity values are depicted in red.