Towards Robust Optical-SAR Object Detection under Missing Modalities: A Dynamic Quality-Aware Fusion Framework

TL;DR

This work tackles robust optical–SAR object detection under missing modalities caused by misalignment and adverse conditions. It introduces QDFNet, a quality-aware fusion framework composed of a Dynamic Modality Quality Assessment (DMQA) module and an Orthogonal Constraint Normalization Fusion (OCNF), with an INN-based preprocessing stage to complete missing data. DMQA quantifies feature reliability through magnitude stability and directional consistency using learnable tokens, while OCNF preserves modality independence and fuses features according to reliability, guided by $R_R$ and $R_S$. Across SpaceNet6-OTD-Fog and OGSOD-2.0, QDFNet achieves state-of-the-art robustness and detection accuracy under missing data, indicating strong practical potential for all-weather remote sensing tasks, with performance improvements observed at varying $MR$ and under both Zero-filling and INN reconstruction strategies.

Abstract

Optical and Synthetic Aperture Radar (SAR) fusion-based object detection has attracted significant research interest in remote sensing, as these modalities provide complementary information for all-weather monitoring. However, practical deployment is severely limited by inherent challenges. Due to distinct imaging mechanisms, temporal asynchrony, and registration difficulties, obtaining well-aligned optical-SAR image pairs remains extremely difficult, frequently resulting in missing or degraded modality data. Although recent approaches have attempted to address this issue, they still suffer from limited robustness to random missing modalities and lack effective mechanisms to ensure consistent performance improvement in fusion-based detection. To address these limitations, we propose a novel Quality-Aware Dynamic Fusion Network (QDFNet) for robust optical-SAR object detection. Our proposed method leverages learnable reference tokens to dynamically assess feature reliability and guide adaptive fusion in the presence of missing modalities. In particular, we design a Dynamic Modality Quality Assessment (DMQA) module that employs learnable reference tokens to iteratively refine feature reliability assessment, enabling precise identification of degraded regions and providing quality guidance for subsequent fusion. Moreover, we develop an Orthogonal Constraint Normalization Fusion (OCNF) module that employs orthogonal constraints to preserve modality independence while dynamically adjusting fusion weights based on reliability scores, effectively suppressing unreliable feature propagation. Extensive experiments on the SpaceNet6-OTD and OGSOD-2.0 datasets demonstrate the superiority and effectiveness of QDFNet compared to state-of-the-art methods, particularly under partial modality corruption or missing data scenarios.

Figures (12)

• Figure 1: Illustration of two missing modality processing approaches and performance comparison of state-of-the-art methods on the SpaceNet6-OTD-Fog dataset 19. (a) Primary optical-SAR detection scenarios under various conditions. (b) Demonstration of two modality completion strategies: Zero (zero-filling for missing modalities) and INN (Inverse Design Network-based reconstruction). (c) Comparison of mAP50 performance for different object detectors (CFT 22, SuperYOLO 16, ICAFusion 25, MMIDet 26, and Ours) on the SpaceNet6-OTD-Fog dataset under increasing modality missing rates (0-0.5).
• Figure 2: Framework of the proposed QDFNet. INN is a pre-trained invertible neural network used to generate randomly missing modalities xing2021decoupling2021. The yellow region represents the DMQA module, which evaluates the reliability of multimodal features by quantifying feature quality across both magnitude and directional dimensions. The orange region denotes the OCNF module, which implements orthogonally constrained fusion of multimodal features and dynamically adjusts the fusion weights of different modalities based on the quality assessment provided by DMQA.
• Figure 3: Overall workflow for the DMQA block which adaptively evaluates feature reliability through iterative token-based interaction by jointly measuring magnitude stability and directional consistency, providing fine-grained spatial reliability maps to guide quality-aware fusion.
• Figure 4: Overall workflow for the OCNF module enforces feature independence through orthogonal weight normalization and integrates quality-aware weighting guided by reliability scores to fuse features, preventing the propagation of low-quality features and noise.
• Figure 5: Visualization of the learning attention map using Grad-CAM under the scenario of MR = 0.3 and Zero-filling. (a) RGB GT Labels. (b) SAR GT Labels. (c) Baseline. (d) w/ DMQA. (e) w/ OCNF. (f) Ours.