Pixel-Wise Multimodal Contrastive Learning for Remote Sensing Images

TL;DR

The paper tackles label scarcity in Earth observation by introducing Pixel-wise Multimodal Contrastive Learning (PIMC), which aligns pixel-wise 2D time-series representations derived from vegetation indices with remote sensing imagery in a CLIP-like self-supervised framework. It constructs 2D recurrence plots from NDVI, EVI, and SAVI across patches and samples, then trains dual encoders to learn a shared latent space that benefits downstream pixel-level forecasting, pixel classification, and land-cover classification. Across PASTIS and EuroSAT, PIMC demonstrates that 2D representations and multimodal contrastive training yield superior or competitive results compared to 1D baselines and other state-of-the-art SSL methods, while enabling robust feature spaces and transfer learning. The approach provides a robust, extensible framework for processing both SITS and RSI modalities and has potential for broader applications such as anomaly detection and multi-modal EO tasks.

Abstract

Satellites continuously generate massive volumes of data, particularly for Earth observation, including satellite image time series (SITS). However, most deep learning models are designed to process either entire images or complete time series sequences to extract meaningful features for downstream tasks. In this study, we propose a novel multimodal approach that leverages pixel-wise two-dimensional (2D) representations to encode visual property variations from SITS more effectively. Specifically, we generate recurrence plots from pixel-based vegetation index time series (NDVI, EVI, and SAVI) as an alternative to using raw pixel values, creating more informative representations. Additionally, we introduce PIxel-wise Multimodal Contrastive (PIMC), a new multimodal self-supervision approach that produces effective encoders based on two-dimensional pixel time series representations and remote sensing imagery (RSI). To validate our approach, we assess its performance on three downstream tasks: pixel-level forecasting and classification using the PASTIS dataset, and land cover classification on the EuroSAT dataset. Moreover, we compare our results to state-of-the-art (SOTA) methods on all downstream tasks. Our experimental results show that the use of 2D representations significantly enhances feature extraction from SITS, while contrastive learning improves the quality of representations for both pixel time series and RSI. These findings suggest that our multimodal method outperforms existing models in various Earth observation tasks, establishing it as a robust self-supervision framework for processing both SITS and RSI. Code avaliable on

Figures (9)

• Figure 1: The figure illustrates the pipeline of our multimodal self-supervision learning approach, wherein we start the process utilizing the patches from the SITS, where pixels are selected to construct the time series using the vegetation indices values. Subsequently, we extract the two-dimensional representation from the time series. Both modalities are employed in the training of the image encoder and time series encoder via the PIMC approach. The trained models and their fine-tuned versions are utilized in downstream tasks to assess the effectiveness of utilizing two-dimensional representations in training via PIMC.
• Figure 2: Illustration of the creation of the 2D representation of pixel-wise vegetation indices from the SITS. (a) Process of dividing the SITS into patches; (b) process for computing the vegetation indices for each SITS patch; (c) selection of pixels within each image patch; (d) construction of 1D time series, representing the vegetation indices of each pixel at different timestamps; and (e) computation of recurrence plot, transforming the 1D time series into a 2D representation that encodes recurrent states in the time series.
• Figure 3: Process of contrastive learning: (a) an RSI patch and (b) the combined 2D recurrence plots of the vegetation indices from one pixel within the RSI patch; and (c) the similarity matrix generated by the dot product between $\mathbf{I}$ and $\mathbf{T}$.
• Figure 4: Illustration of the validation process for encoders generated using PIMC. The RSI images were processed as SITS patches and used as input for baseline encoders and the PIMC image encoder for the land cover classification downstream task. On the other hand, the time series data (1D and 2D representations) were used as input for the raw time series encoder and 2D time series encoder as (baselines), and for the PIMC time series encoder. In this case, two downstream tasks are considered: time series-based vegetation index forecasting and time series-based pixel classification.
• Figure 5: Confusion matrices for the PIMC$^{\text{FT}}$, DINO$^{\text{FT}}$ and SeCo$^{\text{FT}}$ classification models. (a) PIMC fine-tuned; (b) DINO MC fine-tuned; (c) SeCo fine-tuned. The PIMC model shows reduced confusion between classes.