Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration

TL;DR

This work tackles the practical EO challenge of missing modalities at inference by proposing a general multi-modal co-learning framework that operates across tasks. It builds modality-dedicated encoders producing three feature spaces—shared, specific, and unused—guided by a combination of main, auxiliary, contrastive, and modality-discriminant losses, enabling accurate per-modality predictions even when only one modality is available at test time. Across four EO benchmarks spanning binary, multi-class, multi-label classification, and regression, the approach consistently improves single-modality performance over strong baselines and several state-of-the-art methods, demonstrating robust cross-modal knowledge transfer. The method’s flexibility, task-agnostic design, and demonstrated gains substantiate its potential to enhance EO analysis under real-world sensor constraints, with future work extending to more modalities and broader benchmarks.

Abstract

Multi-modal co-learning is emerging as an effective paradigm in machine learning, enabling models to collaboratively learn from different modalities to enhance single-modality predictions. Earth Observation (EO) represents a quintessential domain for multi-modal data analysis, wherein diverse remote sensors collect data to sense our planet. This unprecedented volume of data introduces novel challenges. Specifically, the access to the same sensor modalities at both training and inference stages becomes increasingly complex based on real-world constraints affecting remote sensing platforms. In this context, multi-modal co-learning presents a promising strategy to leverage the vast amount of sensor-derived data available at the training stage to improve single-modality models for inference-time deployment. Most current research efforts focus on designing customized solutions for either particular downstream tasks or specific modalities available at the inference stage. To address this, we propose a novel multi-modal co-learning framework capable of generalizing across various tasks without targeting a specific modality for inference. Our approach combines contrastive and modality discriminative learning together to guide single-modality models to structure the internal model manifold into modality-shared and modality-specific information. We evaluate our framework on four EO benchmarks spanning classification and regression tasks across different sensor modalities, where only one of the modalities available during training is accessible at inference time. Our results demonstrate consistent predictive improvements over state-of-the-art approaches from the recent machine learning and computer vision literature, as well as EO-specific methods. The obtained findings validate our framework in the single-modality inference scenarios across a diverse range of EO applications.

Figures (13)

• Figure 1: Illustration of multi-modal co-learning. The multi-modal data is available for training, but single-modality data is accessible for inference as an all-but-one missing modality scenario.
• Figure 2: Illustration of our framework with shared, specific, and unused features. The main prediction per modality is illustrated in regular arrows, while the losses are shown with dashed lines. Two sensor modalities are shown: an optical image and a radar sits.
• Figure 3: Predictive performance by using different encoder architectures in the framework. Results are from the dataset (regression).
• Figure 4: t-SNE projection of the learned features by the Individual method and our (concatenation of shared and specific features) on the dataset.
• Figure 5: Individual loss functions of our framework across the training. The average across folds and multiple runs is shown for each loss function.