Transformer-Based Dual-Optical Attention Fusion Crowd Head Point Counting and Localization Network

TL;DR

This work tackles accurate head-point crowd counting in challenging UAV scenarios by introducing TAPNet, a bimodal framework that fuses RGB and infrared information through a dual-optical attention fusion module (DAFP) and an adaptive two-branch fusion module (AFDF) to address misalignment. It combines point-based counting with auxiliary point guidance and a spatial random offset augmentation to improve localization and robustness, supported by a composite loss that enforces cross-modal consistency and precise matching. The approach achieves state-of-the-art performance on DroneRGBT and GAII C2 datasets, especially in dense, low-light scenes, and provides code for reproducibility. These contributions advance reliable crowd counting and localization in real-world aerial surveillance tasks where modality variation and misalignment are significant challenges.

Abstract

In this paper, the dual-optical attention fusion crowd head point counting model (TAPNet) is proposed to address the problem of the difficulty of accurate counting in complex scenes such as crowd dense occlusion and low light in crowd counting tasks under UAV view. The model designs a dual-optical attention fusion module (DAFP) by introducing complementary information from infrared images to improve the accuracy and robustness of all-day crowd counting. In order to fully utilize different modal information and solve the problem of inaccurate localization caused by systematic misalignment between image pairs, this paper also proposes an adaptive two-optical feature decomposition fusion module (AFDF). In addition, we optimize the training strategy to improve the model robustness through spatial random offset data augmentation. Experiments on two challenging public datasets, DroneRGBT and GAIIC2, show that the proposed method outperforms existing techniques in terms of performance, especially in challenging dense low-light scenes. Code is available at https://github.com/zz-zik/TAPNet

Figures (3)

• Figure 1: Examples of infrared and visible images. (a) The two rows of people on the left are almost invisible in the visible spectrum under low light conditions, illustrating the fact that IR images are more advantageous in low light conditions. (b) Example of RGB-TIR modal misalignment, showing that the modal misalignment problem is more prominent in target detection from the UAV viewpoint, where the yellow and red boxes denote the annotations of the same objects in the TIR image and the RGB image, respectively.
• Figure 2: Overall architecture of TAPNet. We first extract the image feature representation $\left\{ {{F_{R1}},...,{F_{R4}}} \right\}$ and $\left\{ {{F_{T1}},...,{F_{T4}}} \right\}$ separately using the ResNet50 backbone. Then, a bi-optical attention fusion module is applied to the last two layers of features to fuse the features. Subsequently, the fused two layers of features $\left\{ {{F_3},{F_4}}\right\}$ are passed through an adaptive spatial pyramid pooling (ASPP) module and implicit feature interpolation (IFI), respectively. Finally, these features are cascaded and passed to a regression and classification module to obtain the coordinates and confidence of the final target head point, including “unoccupied” or “occupied” and its probability and localization.
• Figure 3: Adaptive Fusion Architecture Diagram. The module consists of an encoder and decoder, respectively, and a domain-adaptive layer structure based on hybrid kernel functions. The difference with Figure \ref{['fig:pipeline1']} lies in the fact that the Adaptive Fusion Module for Bi-Optical Feature Decomposition (AFD) employs early fusion, which also means that the fused image is passed through BackBone once to obtain the last two layers of the feature map.