Multi-modal Uncertainty Robust Tree Cover Segmentation For High-Resolution Remote Sensing Images

TL;DR

This work tackles cross-modal uncertainty in high-resolution remote sensing for tree-cover segmentation by addressing temporal misalignment across modalities. It introduces MURTreeFormer, a multi-modal segmentation framework with SURM for selective patch-level reconstruction, a cross-modal distillation module for feature alignment, and a tree-cover refinement decoder featuring gradient magnitude attention and a shallow refinement head to preserve fine-grained boundaries. Across Zurich and Shanghai datasets, the approach achieves state-of-the-art performance, notably improving mIoU, IoU, and F1 scores while demonstrating robustness to cross-modal inconsistencies. The method offers a practical solution for reliable, high-resolution tree-cover mapping in real-world, temporally diverse multi-modal data, with potential applications in urban planning and ecological monitoring.

Abstract

Recent advances in semantic segmentation of multi-modal remote sensing images have significantly improved the accuracy of tree cover mapping, supporting applications in urban planning, forest monitoring, and ecological assessment. Integrating data from multiple modalities-such as optical imagery, light detection and ranging (LiDAR), and synthetic aperture radar (SAR)-has shown superior performance over single-modality methods. However, these data are often acquired days or even months apart, during which various changes may occur, such as vegetation disturbances (e.g., logging, and wildfires) and variations in imaging quality. Such temporal misalignments introduce cross-modal uncertainty, especially in high-resolution imagery, which can severely degrade segmentation accuracy. To address this challenge, we propose MURTreeFormer, a novel multi-modal segmentation framework that mitigates and leverages aleatoric uncertainty for robust tree cover mapping. MURTreeFormer treats one modality as primary and others as auxiliary, explicitly modeling patch-level uncertainty in the auxiliary modalities via a probabilistic latent representation. Uncertain patches are identified and reconstructed from the primary modality's distribution through a VAE-based resampling mechanism, producing enhanced auxiliary features for fusion. In the decoder, a gradient magnitude attention (GMA) module and a lightweight refinement head (RH) are further integrated to guide attention toward tree-like structures and to preserve fine-grained spatial details. Extensive experiments on multi-modal datasets from Shanghai and Zurich demonstrate that MURTreeFormer significantly improves segmentation performance and effectively reduces the impact of temporally induced aleatoric uncertainty.

Figures (13)

• Figure 1: The overall architecture of MURTreeFormer. MURTreeFormer contains a selective uncertainty-guided reconstruction module (SURM), a Swin Transformer based encoder with a cross-model distillation module (CDM), and a tree cover super refinement decoder including grad magnitude attention (GMA) and refinement head (RH).
• Figure 2: Illustration of the patch-wise reconstruction module (PatchRecon).
• Figure 3: Illustration of the decoder block. Each stage consists of an AlignUnit and a DecoderUnit. The AlignUnit is responsible for reducing the channel dimension and aligning cross-modal features, while the DecoderUnit is applied to progressively restore fine-grained segmentation details.
• Figure 4: Visual illustration of gradient-based attention maps generated by the proposed gradient magnitude attention (GMA). Two sets of input examples are shown, covering ORS, DSM, and SAR modalities. GMA can capture tree cover boundaries with high spatial precision across modalities, producing visually consistent attention responses despite large spectral and structural differences.
• Figure 5: Comparison of decoder heads. (a) Single-step upsampling head commonly used in Transformer-based segmentation models such as SegFormer. (b) Auxiliary head, typically used during training for deep supervision. (c) The proposed shallow feature refinement head (RH), which performs progressive upsampling and jointly predicts segmentation and edge maps to enhance spatial consistency.