Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data

Lekang Wen, Liang Liao, Jing Xiao, Mi Wang

TL;DR

The Semantic-Guided Modality-Aware (SGMA) framework is proposed, which ensures balanced multimodal learning while reducing intra-class variation and reconciling cross-modal inconsistencies through semantic guidance.

Abstract

Multimodal semantic segmentation integrates complementary information from diverse sensors for remote sensing Earth observation. However, practical systems often encounter missing modalities due to sensor failures or incomplete coverage, termed Incomplete Multimodal Semantic Segmentation (IMSS). IMSS faces three key challenges: (1) multimodal imbalance, where dominant modalities suppress fragile ones; (2) intra-class variation in scale, shape, and orientation across modalities; and (3) cross-modal heterogeneity with conflicting cues producing inconsistent semantic responses. Existing methods rely on contrastive learning or joint optimization, which risk over-alignment, discarding modality-specific cues or imbalanced training, favoring robust modalities, while largely overlooking intra-class variation and cross-modal heterogeneity. To address these limitations, we propose the Semantic-Guided Modality-Aware (SGMA) framework, which ensures balanced multimodal learning while reducing intra-class variation and reconciling cross-modal inconsistencies through semantic guidance. SGMA introduces two complementary plug-and-play modules: (1) Semantic-Guided Fusion (SGF) module extracts multi-scale, class-wise semantic prototypes that capture consistent categorical representations across modalities, estimates per-modality robustness based on prototype-feature alignment, and performs adaptive fusion weighted by robustness scores to mitigate intra-class variation and cross-modal heterogeneity; (2) Modality-Aware Sampling (MAS) module leverages robustness estimations from SGF to dynamically reweight training samples, prioritizing challenging samples from fragile modalities to address modality imbalance. Extensive experiments across multiple datasets and backbones demonstrate that SGMA consistently outperforms state-of-the-art methods, with particularly significant improvements in fragile modalities.

SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data

TL;DR

The Semantic-Guided Modality-Aware (SGMA) framework is proposed, which ensures balanced multimodal learning while reducing intra-class variation and reconciling cross-modal inconsistencies through semantic guidance.

Abstract

Multimodal semantic segmentation integrates complementary information from diverse sensors for remote sensing Earth observation. However, practical systems often encounter missing modalities due to sensor failures or incomplete coverage, termed Incomplete Multimodal Semantic Segmentation (IMSS). IMSS faces three key challenges: (1) multimodal imbalance, where dominant modalities suppress fragile ones; (2) intra-class variation in scale, shape, and orientation across modalities; and (3) cross-modal heterogeneity with conflicting cues producing inconsistent semantic responses. Existing methods rely on contrastive learning or joint optimization, which risk over-alignment, discarding modality-specific cues or imbalanced training, favoring robust modalities, while largely overlooking intra-class variation and cross-modal heterogeneity. To address these limitations, we propose the Semantic-Guided Modality-Aware (SGMA) framework, which ensures balanced multimodal learning while reducing intra-class variation and reconciling cross-modal inconsistencies through semantic guidance. SGMA introduces two complementary plug-and-play modules: (1) Semantic-Guided Fusion (SGF) module extracts multi-scale, class-wise semantic prototypes that capture consistent categorical representations across modalities, estimates per-modality robustness based on prototype-feature alignment, and performs adaptive fusion weighted by robustness scores to mitigate intra-class variation and cross-modal heterogeneity; (2) Modality-Aware Sampling (MAS) module leverages robustness estimations from SGF to dynamically reweight training samples, prioritizing challenging samples from fragile modalities to address modality imbalance. Extensive experiments across multiple datasets and backbones demonstrate that SGMA consistently outperforms state-of-the-art methods, with particularly significant improvements in fragile modalities.
Paper Structure (33 sections, 14 equations, 12 figures, 9 tables, 1 algorithm)

This paper contains 33 sections, 14 equations, 12 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Learning
  4. Incomplete Multimodal Learning
  5. Methodology
  6. Problem Formulation of IMSS
  7. Framework Overview
  8. Semantic-Guided Fusion Module
  9. Modality-aware Sampling Module
  10. Objective Functions
  11. Experiments
  12. Experimental Setup
  13. Datasets
  14. Baseline Methods
  15. Metircs
  16. ...and 18 more sections

Figures (12)

  • Figure 1: Illustration of the challenges of (a) substantial intra-class variation and (b) strong cross-modal heterogeneity in incomplete multimodal semantic segmentation for remote sensing. (a) The same semantic category, building, exhibits large differences in scale and orientation, which complicates the extraction of consistent category features. (b) Different modalities produce conflicting cues: in RGB imagery, roof and ground share similar colors but differ in DSM height, while ground and grass have similar heights in DSM but distinct colors in RGB, making semantic correspondence challenging.
  • Figure 2: Overall framework of the proposed SGMA, which includes the Semantic-Guided Fusion (SGF) and Modality-Aware Sampling (MAS) modules.
  • Figure 3: Structure of the Modality-specific Projector (MP).
  • Figure 4: Qualitative comparisons on the ISPRS under the (a) normal condition and (b) complementary condition.
  • Figure 5: Qualitative comparisons on the DFC2023 under the (a) normal condition and (b) complementary condition.
  • ...and 7 more figures