Prithvi-Complimentary Adaptive Fusion Encoder (CAFE): unlocking full-potential for flood inundation mapping

TL;DR

Flood inundation mapping with Geo-Foundation Models remains challenging due to insufficient local detail capture. Prithvi-CAFE blends a Prithvi pretrained transformer encoder with a CNN residual-CAM path, using adapter-based fine-tuning and a multi-scale, multi-level attention fusion to combine long-range dependencies with fine-grained spatial cues. The approach achieves state-of-the-art results on Sen1Floods11 and FloodPlanet, outperforming baselines including U-Net and large GFMs while using far fewer trainable parameters. This work demonstrates a practical path to leveraging multi-channel remote-sensing data through channel-wise fusion, with implications for efficient, accurate flood mapping in real-world applications.

Abstract

Geo-Foundation Models (GFMs), have proven effective in diverse downstream applications, including semantic segmentation, classification, and regression tasks. However, in case of flood mapping using Sen1Flood11 dataset as a downstream task, GFMs struggles to outperform the baseline U-Net, highlighting model's limitation in capturing critical local nuances. To address this, we present the Prithvi-Complementary Adaptive Fusion Encoder (CAFE), which integrate Prithvi GFM pretrained encoder with a parallel CNN residual branch enhanced by Convolutional Attention Modules (CAM). Prithvi-CAFE enables fast and efficient fine-tuning through adapters in Prithvi and performs multi-scale, multi-level fusion with CNN features, capturing critical local details while preserving long-range dependencies. We achieve state-of-the-art results on two comprehensive flood mapping datasets: Sen1Flood11 and FloodPlanet. On Sen1Flood11 test data, Prithvi-CAFE (IoU 83.41) outperforms the original Prithvi (IoU 82.50) and other major GFMs (TerraMind 82.90, DOFA 81.54, spectralGPT: 81.02). The improvement is even more pronounced on the hold-out test site, where Prithvi-CAFE achieves an IoU of 81.37 compared to the baseline U-Net (70.57) and original Prithvi (72.42). On FloodPlanet, Prithvi-CAFE also surpasses the baseline U-Net and other GFMs, achieving an IoU of 64.70 compared to U-Net (60.14), Terramind (62.33), DOFA (59.15) and Prithvi 2.0 (61.91). Our proposed simple yet effective Prithvi-CAFE demonstrates strong potential for improving segmentation tasks where multi-channel and multi-modal data provide complementary information and local details are critical. The code is released on \href{https://github.com/Sk-2103/Prithvi-CAFE}{Prithvi-CAFE Github}

Figures (6)

• Figure 1: The proposed Prithvi-CAFE architecture. (a) The Spectral Selection module divides input images into two spectral branches. (b) The encoder processes these branches using adapted Prithvi blocks and CNN blocks, respectively. (c) The Multi-Scale Multi-Level Feature Attention Fusion (M$^{2}$FAF) module merges features from both streams. (d) The Decoder integrates them via pyramid pooling and lateral connections to generate the segmentation mask. (e) Adapter modules are attached to each ViT block of Prithvi, as shown in (f). (g) Residual Blocks and (h) Convolutional Attention Modules (CAM) are used to enhance CNN features.
• Figure 2: The effect of bias factor $\beta$ on Transformer (semantic) and CNN (spatial) feature fusion
• Figure 3: Box plot showing distribution of mIoU per image in four fold cross validation on FLoodPlanet
• Figure 4: Comparative visual analysis of the proposed Prithvi-CAFE against other models using FloodPlanet data. Numbers in the lower-right corner indicate mIoU.
• Figure 5: Visualization of feature embedding of FloodPlanet data using t-SNE plots. (a) fully fine-tune Prithvi 2.0 encoder (600M), (b) Adaptive Prithvi fine-tuning (only 45.5M parameters are trained) (c) fully fine-tune TerraMind-Base encoder, (e) fully fine-tune DOFA-Base encoder.