SWIR-LightFusion: Multi-spectral Semantic Fusion of Synthetic SWIR with Thermal IR (LWIR/MWIR) and RGB

TL;DR

This work tackles the challenge of robust scene understanding under adverse conditions by introducing SynSWIR, a synthetic SWIR-like cue generated from LWIR via CLAHE, and integrating it with RGB and MWIR in a tri-modal fusion framework called SWIR-LightFusion. The network uses three modality-specific encoders with lightweight gradient residual blocks, a softmax-gated tri-modal fusion head, and a decoder to produce high-quality fused images in real time. A triplet semantic-consistency loss and extensive benchmarking on RGB–LWIR datasets (and a private RGB–MWIR–SWIR dataset) demonstrate superior fusion quality (contrast, edge definition, structural fidelity) and computational efficiency compared with state-of-the-art baselines, including traditional trimodal methods. The approach significantly advances practical fusion for surveillance and autonomous systems, enabling improved object visibility and scene interpretation where real SWIR data are scarce.

Abstract

Enhancing scene understanding in adverse visibility conditions remains a critical challenge for surveillance and autonomous navigation systems. Conventional imaging modalities, such as RGB and thermal infrared (MWIR / LWIR), when fused, often struggle to deliver comprehensive scene information, particularly under conditions of atmospheric interference or inadequate illumination. To address these limitations, Short-Wave Infrared (SWIR) imaging has emerged as a promising modality due to its ability to penetrate atmospheric disturbances and differentiate materials with improved clarity. However, the advancement and widespread implementation of SWIR-based systems face significant hurdles, primarily due to the scarcity of publicly accessible SWIR datasets. In response to this challenge, our research introduces an approach to synthetically generate SWIR-like structural/contrast cues (without claiming spectral reproduction) images from existing LWIR data using advanced contrast enhancement techniques. We then propose a multimodal fusion framework integrating synthetic SWIR, LWIR, and RGB modalities, employing an optimized encoder-decoder neural network architecture with modality-specific encoders and a softmax-gated fusion head. Comprehensive experiments on public RGB-LWIR benchmarks (M3FD, TNO, CAMEL, MSRS, RoadScene) and an additional private real RGB-MWIR-SWIR dataset demonstrate that our synthetic-SWIR-enhanced fusion framework improves fused-image quality (contrast, edge definition, structural fidelity) while maintaining real-time performance. We also add fair trimodal baselines (LP, LatLRR, GFF) and cascaded trimodal variants of U2Fusion/SwinFusion under a unified protocol. The outcomes highlight substantial potential for real-world applications in surveillance and autonomous systems.

Figures (4)

• Figure 1: The overall structure of the proposed SWIR-LightFusion network. RGB, MWIR, and synthetically generated SWIR images are independently processed by dedicated encoder modules consisting of lightweight gradient residual blocks (Light-GRLB). Extracted features are then seamlessly integrated within the fusion layer, subsequently reconstructed by the decoder module to generate a high-quality fused output image.
• Figure 2: An in-depth explanation of Gradient Residual-based Convolutional Layers hussain2024light, illustrating the integration of pointwise (1x1) convolutions to reduce computational complexity and make the network lightweight while preserving essential gradient features.
• Figure 3: Qualitative results of the proposed SWIR-LightFusion network on the CAMEL, TNO, and M3FD datasets for two selected sequences. For clear comparative visualization, the grayscale images and synthetically generated SWIR images are highlighted in yellow. The regions within bounding boxes clearly illustrate improved structure visibility and object clarity provided by the proposed approach, demonstrating substantial advantages over existing fusion methods.
• Figure 4: Merged radar over all datasets and common metrics (EN, SD, MI, VIF, Qabf, SSIM). For each dataset, scores are normalized per metric across methods and then averaged per metric across M3FD, CAMEL, TNO, RoadScene, and MSRS, yielding a compact and fair comparison. Larger area indicates stronger all-round performance across benchmarks.