STARS: Shared-specific Translation and Alignment for missing-modality Remote Sensing Semantic Segmentation

TL;DR

STARS targets missing-modality semantic segmentation in remote sensing by coupling an asymmetric cross-modality alignment with a pixel-level semantic balanced sampling strategy. The framework uses a shared encoder plus modality-specific encoders, a fusion module, and three FPN decoders to maintain semantic consistency while preserving modality-specific details. Key contributions include (1) a stop-gradient-based asymmetric translation mechanism to prevent feature collapse, and (2) a PSA strategy that balances semantic classes during cross-modality alignment, improving minority-class recognition. Extensive experiments on EarthMiss, WHU-OPT-SAR, and ISPRS Potsdam demonstrate superior performance under incomplete modality conditions and provide insights into feature disentanglement and cross-modality learning dynamics.

Abstract

Multimodal remote sensing technology significantly enhances the understanding of surface semantics by integrating heterogeneous data such as optical images, Synthetic Aperture Radar (SAR), and Digital Surface Models (DSM). However, in practical applications, the missing of modality data (e.g., optical or DSM) is a common and severe challenge, which leads to performance decline in traditional multimodal fusion models. Existing methods for addressing missing modalities still face limitations, including feature collapse and overly generalized recovered features. To address these issues, we propose \textbf{STARS} (\textbf{S}hared-specific \textbf{T}ranslation and \textbf{A}lignment for missing-modality \textbf{R}emote \textbf{S}ensing), a robust semantic segmentation framework for incomplete multimodal inputs. STARS is built on two key designs. First, we introduce an asymmetric alignment mechanism with bidirectional translation and stop-gradient, which effectively prevents feature collapse and reduces sensitivity to hyperparameters. Second, we propose a Pixel-level Semantic sampling Alignment (PSA) strategy that combines class-balanced pixel sampling with cross-modality semantic alignment loss, to mitigate alignment failures caused by severe class imbalance and improve minority-class recognition.

Figures (12)

• Figure 1: Comparison of existing methods for handling missing modalities. On the left, methods based on knowledge distillation or mutual learning are shown, which aim to transfer knowledge from complete modalities to the missing scenarios. On the right, methods based on shared-specific feature learning are displayed, which compensate for the missing modality by decoupling features and utilizing shared/meta features ($\mathcal{F}_{meta}$).
• Figure 2: Overview of the STARS (Shared-specific Translation and Alignment for missing-modality Remote Sensing Semantic Segmentation) framework. During training, the architecture employs shared and modality-specific encoders to extract complementary features from multimodal data through cross-modality alignment and fusion. During inference, the learned universal feature representations enable high-precision performance under single-modality constraints.
• Figure 3: Detailed architecture of the Fusion Module. It integrates modality-specific and shared features using SCSE blocks, residual connections, and concatenation to preserve both distinct physical characteristics and common semantic information.
• Figure 4: Structure of the Spatial and Channel Squeeze and Excitation (SCSE) block. It concurrently recalibrates features along the channel and spatial dimensions to enhance representational power.
• Figure 5: Illustration of the Pixel-level Semantic Alignment (PSA) pipeline. This strategy employs an active sampling mechanism to extract class-balanced pixel samples from bimodal features.