DEF-YOLO: Leveraging YOLO for Concealed Weapon Detection in Thermal Imagin

TL;DR

This work addresses concealed weapon detection in thermal imagery by proposing DEF-YOLO, a YOLOv8-based framework augmented with deformable convolutions and focal loss to handle the deformable, low-detail patterns of weapons under clothing. It introduces TICW, a large 6k thermal dataset with four weapon classes to facilitate real-time surveillance while preserving privacy. DEF-YOLO demonstrates state-of-the-art performance on TICW (mAP@0.5 = 98.4, mAP@0.5:0.95 = 70.3) and strong results on Active THz data, highlighting improved localization and robustness to occlusion and low contrast. The approach balances accuracy and speed, making it suitable for 24x7 security applications, and the dataset provides a valuable resource for future thermal-imaging CWD research.

Abstract

Concealed weapon detection aims at detecting weapons hidden beneath a person's clothing or luggage. Various imaging modalities like Millimeter Wave, Microwave, Terahertz, Infrared, etc., are exploited for the concealed weapon detection task. These imaging modalities have their own limitations, such as poor resolution in microwave imaging, privacy concerns in millimeter wave imaging, etc. To provide a real-time, 24 x 7 surveillance, low-cost, and privacy-preserved solution, we opted for thermal imaging in spite of the lack of availability of a benchmark dataset. We propose a novel approach and a dataset for concealed weapon detection in thermal imagery. Our YOLO-based architecture, DEF-YOLO, is built with key enhancements in YOLOv8 tailored to the unique challenges of concealed weapon detection in thermal vision. We adopt deformable convolutions at the SPPF layer to exploit multi-scale features; backbone and neck layers to extract low, mid, and high-level features, enabling DEF-YOLO to adaptively focus on localization around the objects in thermal homogeneous regions, without sacrificing much of the speed and throughput. In addition to these simple yet effective key architectural changes, we introduce a new, large-scale Thermal Imaging Concealed Weapon dataset, TICW, featuring a diverse set of concealed weapons and capturing a wide range of scenarios. To the best of our knowledge, this is the first large-scale contributed dataset for this task. We also incorporate focal loss to address the significant class imbalance inherent in the concealed weapon detection task. The efficacy of the proposed work establishes a new benchmark through extensive experimentation for concealed weapon detection in thermal imagery.

Figures (5)

• Figure 1: Subject in (a) sitting position carrying scissors in right leg, knife in left leg, gun in waist, and knife in arm. (b) standing position with front viewpoint carrying a knife in the right thigh, a gun and a knife in the waist, and scissors in the chest. (c) side viewpoint carrying a cleaver at the waist and scissors in the chest. (d) back viewpoint carrying a gun and a cleaver. (e) front viewpoint carrying a gun in a thin jacket. (f) front viewpoint and less temperature gradient between the weapon, gun, knife concealed in the waist, and the subject's body.
• Figure 2: The proposed DEF-YOLO for CWD with modified modules in red.
• Figure 3: A) Standard convolution using a regular grid is shown in green dots. B) Deformable convolution using an irregular grid in red dots with offset direction. C) Final deformed shape showing improved localization.
• Figure 4: Examples demonstrating the performance with each modification from YOLOv8 to DEF-YOLO.
• Figure 5: Examples showing the performance of YOLOv8 and DEF-YOLO.