Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation

TL;DR

The paper introduces DOFA, a neuroplasticity-inspired, wavelength-conditioned multimodal foundation model for Earth observation that flexibly handles diverse sensors and spectral bands through a dynamic hypernetwork. It jointly trains a shared backbone with a wavelength-aware patch embedding and a dynamic decoder, using masked image modeling and distillation to enable continual multimodal pretraining; DOFA+ further leverages strong priors and hierarchical distillation for efficient domain adaptation. Comprehensive experiments across GEO-Bench, PANGEA, RESISC45, and DIOR demonstrate state-of-the-art performance and robust generalization to unseen sensors and spectral configurations, while achieving notable efficiency gains over prior large-scale EO models. The work provides a scalable pathway toward open-world EO foundation models, with publicly available code and pretrained weights to support deployment and further research.

Abstract

Earth observation (EO) in open-world settings presents a unique challenge: different applications rely on diverse sensor modalities, each with varying ground sampling distances, spectral ranges, and numbers of spectral bands. However, existing EO foundation models are typically tailored to specific sensor types, making them inflexible when generalizing across the heterogeneous landscape of EO data. To address this, we propose the Dynamic One-For-All (DOFA) model, a unified, multimodal foundation framework designed for diverse vision tasks in EO. Inspired by neural plasticity, DOFA utilizes a wavelength-conditioned dynamic hypernetwork to process inputs from five distinct satellite sensors flexibly. By continually pretraining on five EO modalities, DOFA achieves state-of-the-art performance across multiple downstream tasks and generalizes well to unseen modalities. Enhanced with hybrid continual pretraining, DOFA+ requires significantly fewer computational resources while outperforming counterparts trained with extensive GPU budgets. Experiments on diverse datasets highlight DOFA's potential as a foundation for general-purpose vision models in the sensor-diverse EO domain. The code and pre-trained weights are publicly available at https://github.com/zhu-xlab/DOFA.

Figures (8)

• Figure 1: Motivation of DOFA. Our primary purpose is to develop versatile foundation models capable of adaptively processing various EO data modalities.
• Figure 2: Motivation and main architecture of DOFA. We design DOFA to emulate the Neuroplasticity hebb2005organizationzucker2002shortdan2004spike mechanism for processing multimodal EO data. (1) Illustration of the brain's capability to adapt its structure and function to learned information, experience, or injury. (2) Illustration of the core idea: DOFA is designed to adaptively alter its network weights in response to novel data modalities.
• Figure 3: Architecture and training details. DOFA builds on masked image modeling, introducing a significant advancement by processing input images with any number of channels within a single framework.
• Figure 4: Illustration of weight space interpolation for new sensors or a combination of spectral bands.
• Figure 5: Dynamic weight generator and continual training framework. (1) The central wavelengths of each band are utilized to derive weights tailored to each wavelength. (2) Continual pretraining process. There is a distillation and a reconstruction loss.