Robust Pedestrian Detection with Uncertain Modality

TL;DR

This work tackles robust pedestrian detection under uncertain modality availability in RGB-NIR-TIR cross-modal scenarios. It introduces the TRNT dataset to enable research on triplet RGB–NIR–TIR fusion under real-world modality variations, and proposes AUNet, which combines Unified Modality Validation Refinement (UMVR) with a CLIP-guided semantic refinement and a Modality-Aware Interaction (MAI) module to adaptively fuse information from whatever modalities are present. The key contributions are (1) the TRNT dataset with seven input configurations and diverse capture modes, (2) the AUNet architecture featuring UMVR and MAI to handle uncertain inputs without retraining for each configuration, and (3) extensive experiments on TRNT and LLVIP showing state-of-the-art performance and robustness under modality uncertainty. The approach advances practical, 24/7 pedestrian detection for surveillance and autonomous systems by enabling reliable fusion across flexible modality combinations and maintaining real-time performance.

Abstract

Existing cross-modal pedestrian detection (CMPD) employs complementary information from RGB and thermal-infrared (TIR) modalities to detect pedestrians in 24h-surveillance systems.RGB captures rich pedestrian details under daylight, while TIR excels at night. However, TIR focuses primarily on the person's silhouette, neglecting critical texture details essential for detection. While the near-infrared (NIR) captures texture under low-light conditions, which effectively alleviates performance issues of RGB and detail loss in TIR, thereby reducing missed detections. To this end, we construct a new Triplet RGB-NIR-TIR (TRNT) dataset, comprising 8,281 pixel-aligned image triplets, establishing a comprehensive foundation for algorithmic research. However, due to the variable nature of real-world scenarios, imaging devices may not always capture all three modalities simultaneously. This results in input data with unpredictable combinations of modal types, which challenge existing CMPD methods that fail to extract robust pedestrian information under arbitrary input combinations, leading to significant performance degradation. To address these challenges, we propose the Adaptive Uncertainty-aware Network (AUNet) for accurately discriminating modal availability and fully utilizing the available information under uncertain inputs. Specifically, we introduce Unified Modality Validation Refinement (UMVR), which includes an uncertainty-aware router to validate modal availability and a semantic refinement to ensure the reliability of information within the modality. Furthermore, we design a Modality-Aware Interaction (MAI) module to adaptively activate or deactivate its internal interaction mechanisms per UMVR output, enabling effective complementary information fusion from available modalities.

Figures (4)

• Figure 1: Illustration of our idea. (a) demonstrates the complementary benefits of the NIR for RGB and TIR. (b) illustrates the performance degradation of existing CMPD methods under single-modal and other cross-modal inputs. we conduct evaluation using ICAFusion shen2024icafusion and our AUNet trained on our DRNT.
• Figure 2: Data Diversity. The scenarios and examples of our benchmark TRNT. TRNT covers diverse scenarios with various illumination, perspective, occlusion, and season.
• Figure 3: Illustration of our AUNet, including two main components: 1) Unified Modality Validation Refinement (UMVR), which integrates an Uncertainty-aware Router (UAR) validation and CLIP-driven Semantic Refinement (CSR); 2) Modality-Aware Interaction (MAI), which effectively integrates complementary information from the available modalities. This figure depicts a scenario where all three modalities are available, corresponding to the UMVR output of [1, 1, 1].
• Figure 4: Qualitative Comparison of INSANet, TINet, DE-YOLO, and our AUNet on the TRNT dataset. Red boxes indicate detection results. Blue circles highlight pedestrians missed. Orange circles mark the location of an incorrect detection. Yellow circles represent overlapping multiple detection boxes for the same instance.