Spatial Context Improves the Integration of Text with Remote Sensing for Mapping Environmental Variables

TL;DR

This work tackles the challenge of mapping environmental variables by fusing high-resolution aerial imagery with geolocated text from sources like Wikipedia. It introduces a vision-language transformer architecture that encodes images and text, incorporates a learnable geolocation encoding, and uses an attention-based fusion module to weigh information from nearby observations, thus exploiting spatial context. On EcoWikiRS and SWECO25, the approach consistently outperforms single-location and unimodal baselines, with notable gains for climatic, edaphic, population, and land-use variables, while vegetation remains image-dominated. The study provides a scalable framework for integrating sparse textual data into spatial regression tasks and highlights the conditions under which text contributes most effectively, offering practical implications for ecological mapping and beyond.

Abstract

Recent developments in natural language processing highlight text as an emerging data source for ecology. Textual resources carry unique information that can be used in complementarity with geospatial data sources, thus providing insights at the local scale into environmental conditions and properties hidden from more traditional data sources. Leveraging textual information in a spatial context presents several challenges. First, the contribution of textual data remains poorly defined in an ecological context, and it is unclear for which tasks it should be incorporated. Unlike ubiquitous satellite imagery or environmental covariates, the availability of textual data is sparse and irregular; its integration with geospatial data is not straightforward. In response to these challenges, this work proposes an attention-based approach that combines aerial imagery and geolocated text within a spatial neighbourhood, i.e. integrating contributions from several nearby observations. Our approach combines vision and text representations with a geolocation encoding, with an attention-based module that dynamically selects spatial neighbours that are useful for predictive tasks.The proposed approach is applied to the EcoWikiRS dataset, which combines high-resolution aerial imagery with sentences extracted from Wikipedia describing local environmental conditions across Switzerland. Our model is evaluated on the task of predicting 103 environmental variables from the SWECO25 data cube. Our approach consistently outperforms single-location or unimodal, i.e. image-only or text-only, baselines. When analysing variables by thematic groups, results show a significant improvement in performance for climatic, edaphic, population and land use/land cover variables, underscoring the benefit of including the spatial context when combining text and image data.

Figures (13)

• Figure 1: We study how to effectively combine remote sensing images and geolocated text for predicting a variety of environmental variables with a single deep learning model. For a location of interest ($x_0$), we use the geolocation of species observations in crowdsourcing platforms as a way to introduce geolocated text: We combine aerial images ($I_0$ to $I_3$) with text describing the species habitat extracted from its Wikipedia article ($T_0$ to $T_3$) and encode them with pretrained vision and text encoders, respectively. We incorporate several spatial neighbours that are in the vicinity of the location of interest and keep the spatial organisation of the observation by encoding geolocation ($x_0$ to $x_3$). We fuse all the textual and visual inputs through an attention-based fusion module that dynamically selects which neighbours are useful.
• Figure 2: The proposed model architecture is illustrated with a spatial context of 3 nearest neighbours (k=3), with images, text and their respective geolocations as inputs. The pretrained encoders for image and text are frozen, while the location encoder and the attention-based fusion module are trained. Random token masking (dashed arrows) is applied on the generated visual ($v_i$) and textual ($t_{i,j}$) embeddings after the fusion with the location encoding ($LOC_i$). The final predictions are derived from the classification token (CLS) at the output of the attention-based fusion module. Colours denote distinct geographic locations.
• Figure 3: Different experimental settings used to explore various modality combinations, using three spatial neighbours (k=$3$) and four sentences per location ($j=4$). Colours denote distinct geographic locations.
• Figure 4: Mean coefficient of correlation $R^2$ on the EcoWikiRS test set for predicting SWECO25variables, with images, text or their combination as input, comparing with or without spatial context (k=10). The black line over each bar indicates the standard deviation across five random seeds.
• Figure 5: Mean coefficient of correlation $R^2$ for predicting SWECO25 variables on the EcoWikiRS test set, when adding incrementally more neighbours. The axis is discontinuous between $0.27$ and $0.40$ for clarity of presentation.