Beyond RGB: Leveraging Vision Transformers for Thermal Weapon Segmentation
Akhila Kambhatla, Ahmed R Khaled
TL;DR
This study investigates the use of Vision Transformers for pixel-level thermal weapon segmentation in low-light environments, a domain where RGB-based methods struggle. It benchmarks four transformer architectures—SegFormer, SegNeXt, DeepLabV3+, and Swin Transformer—on a real-world, SAM2-annotated thermal dataset of 9,711 images, using a unified MMSegmentation pipeline. Results show SegFormer variations achieve the highest accuracy, with SegFormer-b5 reaching 94.15% mIoU and SegFormer-b0 delivering real-time inference at 98.32 FPS, while SegNeXt variants provide favorable speed-accuracy trade-offs. The work demonstrates strong generalization in challenging thermal conditions and highlights the value of domain-specific preprocessing and a BCE+Dice loss to optimize boundary accuracy, proposing SegFormer-b3 as a practical deployment choice and outlining future directions in cross-modal fusion and multiclass weapon segmentation.
Abstract
Thermal weapon segmentation is crucial for surveillance and security applications, enabling robust detection under lowlight and visually obscured conditions where RGB-based systems fail. While convolutional neural networks (CNNs) dominate thermal segmentation literature, their ability to capture long-range dependencies and fine structural details is limited. Vision Transformers (ViTs), with their global context modeling capabilities, have achieved state-of-the-art results in RGB segmentation tasks, yet their potential in thermal weapon segmentation remains underexplored. This work adapts and evaluates four transformer-based architectures SegFormer, DeepLabV3\+, SegNeXt, and Swin Transformer for binary weapon segmentation on a custom thermal dataset comprising 9,711 images collected from real world surveillance videos and automatically annotated using SAM2. We employ standard augmentation strategies within the MMSegmentation framework to ensure robust model training and fair architectural comparison. Experimental results demonstrate significant improvements in segmentation performance: SegFormer-b5 achieves the highest mIoU (94.15\%) and Pixel Accuracy (97.04\%), while SegFormer-b0 provides the fastest inference speed (98.32 FPS) with competitive mIoU (90.84\%). SegNeXt-mscans offers balanced performance with 85.12 FPS and 92.24\% mIoU, and DeepLabV3\+ R101-D8 reaches 92.76\% mIoU at 29.86 FPS. The transformer architectures demonstrate robust generalization capabilities for weapon detection in low-light and occluded thermal environments, with flexible accuracy-speed trade-offs suitable for diverse real-time security applications.