Context-Aware Multimodal Representation Learning for Spatio-Temporally Explicit Environmental Modelling

TL;DR

This work tackles the need for flexible, high-resolution environmental representations by proposing a context-aware, multimodal learning framework that fuses Sentinel-1 and Sentinel-2 into a unified 10 m spatio-temporally explicit latent space. The method uses a two-stage approach: modality-specific autoencoders with context-aware reconstruction, followed by a lightweight fusion stage that yields a 7-dimensional central-pixel embedding, preserving both spatial coherence and temporal fidelity. Qualitative analyses show coherent spatial organization in the learned space, while quantitative results demonstrate that the embeddings support ecologically meaningful tasks, notably forecasting Gross Primary Production (GPP) with a 90-day temporal window. The framework offers a flexible, extensible foundation for downstream environmental modelling across scales and sensors, enabling analysis-ready representations suitable for a wide range of Earth system applications.

Abstract

Earth observation (EO) foundation models have emerged as an effective approach to derive latent representations of the Earth system from various remote sensing sensors. These models produce embeddings that can be used as analysis-ready datasets, enabling the modelling of ecosystem dynamics without extensive sensor-specific preprocessing. However, existing models typically operate at fixed spatial or temporal scales, limiting their use for ecological analyses that require both fine spatial detail and high temporal fidelity. To overcome these limitations, we propose a representation learning framework that integrates different EO modalities into a unified feature space at high spatio-temporal resolution. We introduce the framework using Sentinel-1 and Sentinel-2 data as representative modalities. Our approach produces a latent space at native 10 m resolution and the temporal frequency of cloud-free Sentinel-2 acquisitions. Each sensor is first modeled independently to capture its sensor-specific characteristics. Their representations are then combined into a shared model. This two-stage design enables modality-specific optimisation and easy extension to new sensors, retaining pretrained encoders while retraining only fusion layers. This enables the model to capture complementary remote sensing data and to preserve coherence across space and time. Qualitative analyses reveal that the learned embeddings exhibit high spatial and semantic consistency across heterogeneous landscapes. Quantitative evaluation in modelling Gross Primary Production reveals that they encode ecologically meaningful patterns and retain sufficient temporal fidelity to support fine-scale analyses. Overall, the proposed framework provides a flexible, analysis-ready representation learning approach for environmental applications requiring diverse spatial and temporal resolutions.

Figures (7)

• Figure 1: Spatio-temporal context window used for reconstruction. The autoencoder processes 15×15 pixel patches over 11 time steps, embedding each observation in its environmental context. The central pixel is reconstructed, while neighboring pixels provide context with exponentially decaying loss weights by distance.
• Figure 2: Overview of the Modality Autoencoder. The model processes spatio-temporal input patches of size $15 \times 15$ pixels over 11 temporal frames, using either 10 Sentinel-2 bands or 2 Sentinel-1 bands. It integrates convolutional layers, CNN-Attention modules (Fig. \ref{['fig:multiscale_block']}), transformer-based temporal encoders, and linear layers. At the bottleneck, the spectral information is compressed into a latent representation, from which the decoder reconstructs the original input.
• Figure 3: Multiscale convolution block used in the modality autoencoder. Local and broader spatial context are captured through parallel convolutional paths; features are fused by convolution and channel-attention before being passed to the temporal encoder (Fig. \ref{['fig:modality-encoder']}).
• Figure 4: Multimodal data-fusion architecture combining pretrained Sentinel-1 and Sentinel-2 autoencoders. Latent features from each modality are projected and stacked, processed by a Transformer with temporal positional encodings, and mapped into a shared latent representation.
• Figure 5: Training and validation losses for Sentinel-1 (left column) and Sentinel-2 (middle and right columns) models during context-aware pretraining. The first row shows losses computed for the central pixel only, including MAE and SAM, while the second row depicts corresponding metrics for the spatial surroundings (MAE and SSIM). Both models exhibit stable convergence and consistent performance between training and validation, indicating effective reconstruction of both central and contextual features.