Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fusion Meets Diverse Conditions: A High-diversity Benchmark and Baseline for UAV-based Multimodal Object Detection with Condition Cues

Chen Chen, Kangcheng Bin, Ting Hu, Jiahao Qi, Xingyue Liu, Tianpeng Liu, Zhen Liu, Yongxiang Liu, Ping Zhong

TL;DR

This work tackles robust UAV-based object detection using RGB and IR imagery under highly diverse conditions. It introduces ATR-UMOD, a high-diversity dataset spanning broad altitudes, viewpoints, all-day/all-year timing, weather, illumination, and scenes, annotated with six condition attributes and eleven object categories, and presents PCDF, a prompt-guided, condition-aware fusion framework that leverages sample-specific condition prompts and a decoupling mechanism to dynamically reweight modality contributions. The core contributions are the SCPL module for constructing informative per-sample condition prompts via CLIP, the PCD module that separates condition-related and invariant features with specialized losses, and the CDF module that performs detector-oriented fusion based on condition guidance. Empirical results on ATR-UMOD show PCDF achieving superior performance over state-of-the-art unimodal and multimodal methods, with ablations confirming the importance of each component and the practical impact of condition-aware fusion for real-world UAV perception.

Abstract

Unmanned aerial vehicles (UAV)-based object detection with visible (RGB) and infrared (IR) images facilitates robust around-the-clock detection, driven by advancements in deep learning techniques and the availability of high-quality dataset. However, the existing dataset struggles to fully capture real-world complexity for limited imaging conditions. To this end, we introduce a high-diversity dataset ATR-UMOD covering varying scenarios, spanning altitudes from 80m to 300m, angles from 0° to 75°, and all-day, all-year time variations in rich weather and illumination conditions. Moreover, each RGB-IR image pair is annotated with 6 condition attributes, offering valuable high-level contextual information. To meet the challenge raised by such diverse conditions, we propose a novel prompt-guided condition-aware dynamic fusion (PCDF) to adaptively reassign multimodal contributions by leveraging annotated condition cues. By encoding imaging conditions as text prompts, PCDF effectively models the relationship between conditions and multimodal contributions through a task-specific soft-gating transformation. A prompt-guided condition-decoupling module further ensures the availability in practice without condition annotations. Experiments on ATR-UMOD dataset reveal the effectiveness of PCDF.

Fusion Meets Diverse Conditions: A High-diversity Benchmark and Baseline for UAV-based Multimodal Object Detection with Condition Cues

TL;DR

This work tackles robust UAV-based object detection using RGB and IR imagery under highly diverse conditions. It introduces ATR-UMOD, a high-diversity dataset spanning broad altitudes, viewpoints, all-day/all-year timing, weather, illumination, and scenes, annotated with six condition attributes and eleven object categories, and presents PCDF, a prompt-guided, condition-aware fusion framework that leverages sample-specific condition prompts and a decoupling mechanism to dynamically reweight modality contributions. The core contributions are the SCPL module for constructing informative per-sample condition prompts via CLIP, the PCD module that separates condition-related and invariant features with specialized losses, and the CDF module that performs detector-oriented fusion based on condition guidance. Empirical results on ATR-UMOD show PCDF achieving superior performance over state-of-the-art unimodal and multimodal methods, with ablations confirming the importance of each component and the practical impact of condition-aware fusion for real-world UAV perception.

Abstract

Unmanned aerial vehicles (UAV)-based object detection with visible (RGB) and infrared (IR) images facilitates robust around-the-clock detection, driven by advancements in deep learning techniques and the availability of high-quality dataset. However, the existing dataset struggles to fully capture real-world complexity for limited imaging conditions. To this end, we introduce a high-diversity dataset ATR-UMOD covering varying scenarios, spanning altitudes from 80m to 300m, angles from 0° to 75°, and all-day, all-year time variations in rich weather and illumination conditions. Moreover, each RGB-IR image pair is annotated with 6 condition attributes, offering valuable high-level contextual information. To meet the challenge raised by such diverse conditions, we propose a novel prompt-guided condition-aware dynamic fusion (PCDF) to adaptively reassign multimodal contributions by leveraging annotated condition cues. By encoding imaging conditions as text prompts, PCDF effectively models the relationship between conditions and multimodal contributions through a task-specific soft-gating transformation. A prompt-guided condition-decoupling module further ensures the availability in practice without condition annotations. Experiments on ATR-UMOD dataset reveal the effectiveness of PCDF.
Paper Structure (20 sections, 11 equations, 5 figures, 4 tables)

This paper contains 20 sections, 11 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RGB-IR UOD Dataset
  4. Condition Representation Method
  5. Condition-guided Fusion Method
  6. ATR-UMOD Dataset
  7. Dataset Construction
  8. Dataset Statistics
  9. Advances of ATR-UMOD Dataset
  10. Method
  11. Overview
  12. Sample-specific Condition Prompt Learning
  13. Prompt-guided Condition-decoupling
  14. Condition-aware Dynamic Fusion
  15. Experiments
  16. ...and 5 more sections

Figures (5)

  • Figure 1: High-diversity imaging conditions in our ATR-UMOD. Some representative examples are shown for each condition. (a) Broad Altitude Range: It encompasses an altitude range from 80m to 300m, offering a rich resource for multi-scale object analysis. (b) Extensive Angle Coverage: Nearly full angular coverage from 0° to 75° ensures comprehensive object appearances from various viewpoints. (c) Comprehensive Time Span: All-day data collection captures fluctuations in light, shadows, and thermal characteristics over time. (d) Challenging Weather Conditions: Incorporating 7 typical and extreme weather conditions enhances robustness in real-world applications. (e) Rich Illumination Variations: It covers 6 illumination levels from lightless to high-light, improving adaptability to varying image qualities. (f) Diverse Scenario Types: Considering cross-scene generalization, it spans 11 scenarios types with complex backgrounds. These conditions are additionally annotated in each image pair, providing valuable high-level contextual insights and establishing a comprehensive benchmark for condition-specific performance evaluation.
  • Figure 2: Advantage and challenge in our dataset. (a) Annotation example. (b) Performance degradation on ATR-UMOD.
  • Figure 3: The object and attribute statistics of the ATR-UMOD dataset. Note that CR, SV, VN, BS, FC, TK, ME, TR, ER, CE and TT represent car, SUV, van, bus, freight car, truck, motorcycle, trailer, excavator, crane, and tank truck categories, respectively.
  • Figure 4: Overview structure of the proposed PCDF method.
  • Figure 5: Qualitative comparison on ATR-UMOD dataset. Confidence threshold is 0.25. Fusion method results are displayed on IR images to align with the supervisory labels. Missed and incorrectly detected objects are indicated with blue and red dashed circles, respectivelely.