Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DIS2: Disentanglement Meets Distillation with Classwise Attention for Robust Remote Sensing Segmentation under Missing Modalities

Nhi Kieu, Kien Nguyen, Arnold Wiliem, Clinton Fookes, Sridha Sridharan

TL;DR

DIS2 tackles robust semantic segmentation in remote sensing when one or more modalities are missing. It combines three RS-tailored pillars with a unified DLKD framework, a hierarchical fusion backbone, and a class-aware learning module to actively compensate for unavailable data. The method demonstrates consistent gains over state-of-the-art on Vaihingen and Potsdam, particularly for underrepresented classes and under missing-modality scenarios. This approach offers a practical, scalable solution for deploying reliable RS segmentation systems in real-world, sensor-incomplete environments, with publicly releasable code to foster further research.

Abstract

The efficacy of multimodal learning in remote sensing (RS) is severely undermined by missing modalities. The challenge is exacerbated by the RS highly heterogeneous data and huge scale variation. Consequently, paradigms proven effective in other domains often fail when confronted with these unique data characteristics. Conventional disentanglement learning, which relies on significant feature overlap between modalities (modality-invariant), is insufficient for this heterogeneity. Similarly, knowledge distillation becomes an ill-posed mimicry task where a student fails to focus on the necessary compensatory knowledge, leaving the semantic gap unaddressed. Our work is therefore built upon three pillars uniquely designed for RS: (1) principled missing information compensation, (2) class-specific modality contribution, and (3) multi-resolution feature importance. We propose a novel method DIS2, a new paradigm shifting from modality-shared feature dependence and untargeted imitation to active, guided missing features compensation. Its core novelty lies in a reformulated synergy between disentanglement learning and knowledge distillation, termed DLKD. Compensatory features are explicitly captured which, when fused with the features of the available modality, approximate the ideal fused representation of the full-modality case. To address the class-specific challenge, our Classwise Feature Learning Module (CFLM) adaptively learn discriminative evidence for each target depending on signal availability. Both DLKD and CFLM are supported by a hierarchical hybrid fusion (HF) structure using features across resolutions to strengthen prediction. Extensive experiments validate that our proposed approach significantly outperforms state-of-the-art methods across benchmarks.

DIS2: Disentanglement Meets Distillation with Classwise Attention for Robust Remote Sensing Segmentation under Missing Modalities

TL;DR

DIS2 tackles robust semantic segmentation in remote sensing when one or more modalities are missing. It combines three RS-tailored pillars with a unified DLKD framework, a hierarchical fusion backbone, and a class-aware learning module to actively compensate for unavailable data. The method demonstrates consistent gains over state-of-the-art on Vaihingen and Potsdam, particularly for underrepresented classes and under missing-modality scenarios. This approach offers a practical, scalable solution for deploying reliable RS segmentation systems in real-world, sensor-incomplete environments, with publicly releasable code to foster further research.

Abstract

The efficacy of multimodal learning in remote sensing (RS) is severely undermined by missing modalities. The challenge is exacerbated by the RS highly heterogeneous data and huge scale variation. Consequently, paradigms proven effective in other domains often fail when confronted with these unique data characteristics. Conventional disentanglement learning, which relies on significant feature overlap between modalities (modality-invariant), is insufficient for this heterogeneity. Similarly, knowledge distillation becomes an ill-posed mimicry task where a student fails to focus on the necessary compensatory knowledge, leaving the semantic gap unaddressed. Our work is therefore built upon three pillars uniquely designed for RS: (1) principled missing information compensation, (2) class-specific modality contribution, and (3) multi-resolution feature importance. We propose a novel method DIS2, a new paradigm shifting from modality-shared feature dependence and untargeted imitation to active, guided missing features compensation. Its core novelty lies in a reformulated synergy between disentanglement learning and knowledge distillation, termed DLKD. Compensatory features are explicitly captured which, when fused with the features of the available modality, approximate the ideal fused representation of the full-modality case. To address the class-specific challenge, our Classwise Feature Learning Module (CFLM) adaptively learn discriminative evidence for each target depending on signal availability. Both DLKD and CFLM are supported by a hierarchical hybrid fusion (HF) structure using features across resolutions to strengthen prediction. Extensive experiments validate that our proposed approach significantly outperforms state-of-the-art methods across benchmarks.
Paper Structure (13 sections, 14 equations, 6 figures, 3 tables)

This paper contains 13 sections, 14 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Disentanglement Learning with Knowledge Distillation (DLKD)
  5. Hierarchical Hybrid Fusion (HF)
  6. Classwise Feature Learning Module (CFLM)
  7. Experiments
  8. Experimental Setting
  9. Result Analysis
  10. Quantitative Analysis:
  11. Qualitative Analysis:
  12. Ablation Study:
  13. Conclusion

Figures (6)

  • Figure 1: Illustration of our proposed method
  • Figure 2: Model flow based on scenario at inference time
  • Figure 3: Qualitative results on the Vaihingen and Potsdam datasets under different modality settings. Our method remains robust under missing-modality scenarios, producing segmentation maps that closely match ground truth, with more intact predictions for buildings, roads, and vegetation, and sharper object boundaries for cars. (A) Full Modality, (B) Missing NDSM, and (C) Missing RGIR.
  • Figure 4: Classwise Learnable Query Heatmap
  • Figure 5: Intermediate feature activation maps at 4 pyramid levels
  • ...and 1 more figures