Evaluating an Adaptive Multispectral Turret System for Autonomous Tracking Across Variable Illumination Conditions

TL;DR

This work addresses robust, real-time object detection for autonomous turrets operating under variable illumination by proposing an adaptive RGB--LWIR fusion framework. It systematically trains 33 YOLOv11n detectors across 11 fusion ratios and three illumination conditions, and introduces a lux-based runtime mechanism to select the best condition-specific model. Key contributions include a practical linear fusion scheme with 11 fusion levels, a lighting-aware model specialization strategy, and an empirical demonstration of significant detection gains over single-modality baselines across full-light, dim-light, and no-light settings. The approach enables reliable autonomous tracking in mission-critical scenarios and provides open-source data and code for replication and further research.

Abstract

Autonomous robotic platforms are playing a growing role across the emergency services sector, supporting missions such as search and rescue operations in disaster zones and reconnaissance. However, traditional red-green-blue (RGB) detection pipelines struggle in low-light environments, and thermal-based systems lack color and texture information. To overcome these limitations, we present an adaptive framework that fuses RGB and long-wave infrared (LWIR) video streams at multiple fusion ratios and dynamically selects the optimal detection model for each illumination condition. We trained 33 You Only Look Once (YOLO) models on over 22,000 annotated images spanning three light levels: no-light (<10 lux), dim-light (10-1000 lux), and full-light (>1000 lux). To integrate both modalities, fusion was performed by blending aligned RGB and LWIR frames at eleven ratios, from full RGB (100/0) to full LWIR (0/100) in 10% increments. Evaluation showed that the best full-light model (80/20 RGB-LWIR) and dim-light model (90/10 fusion) achieved 92.8% and 92.0% mean confidence; both significantly outperformed the YOLOv5 nano (YOLOv5n) and YOLOv11 nano (YOLOv11n) baselines. Under no-light conditions, the top 40/60 fusion reached 71.0%, exceeding baselines though not statistically significant. Adaptive RGB-LWIR fusion improved detection confidence and reliability across all illumination conditions, enhancing autonomous robotic vision performance.

Figures (7)

• Figure 1: Example images illustrating the two modalities: RGB image (A) and corresponding LWIR thermal image (B) of the same scene.
• Figure 2: Hardware and system overview of the autonomous turret, illustrating the four independent experimental variables: fusion level, illumination category, model architecture, and target characteristics
• Figure 3: Representative illustration of the eleven fusion levels used for training and evaluation. Columns progress from full RGB (top left) to full thermal (bottom right) in 10% increments.
• Figure 4: Average detection confidence of YOLOv11n models across eleven RGB–LWIR fusion levels under three lighting conditions. Higher values indicate greater model certainty in identifying mug targets. Error bars represent the standard error of the mean (SEM), as mean $\pm$ SEM, calculated as $\sigma / \sqrt{n}$.
• Figure 5: Heatmap visualization of mean detection confidence across RGB--LWIR fusion ratios and mug colors, separated by lighting regime. Each panel corresponds to one illumination condition (full-light, dim-light, no-light), with color indicating mean confidence and numeric labels showing exact values. The quintile-based color scale (Q1--Q5) highlights relative performance tiers, enabling direct comparison across panels.