Perceptual Region-Driven Infrared-Visible Co-Fusion for Extreme Scene Enhancement

TL;DR

The paper tackles the challenge of fusing infrared and visible imagery under extreme conditions while preserving visible geometric features and incorporating thermal data. It presents a region-perception-driven framework that jointly handles multi-exposure and multi-modal data using a space-variant exposure (SVE) camera, with region-guided registration and adaptive, SSIM-informed fusion. Key contributions include a space-variant hardware design, a regional perception map with BI/WC/CF/V features, an adaptive feature-merging strategy, a region-aware multi-exposure fusion, and an infrared-visible complementary fusion module. Extensive experiments on rocket-launch and public datasets demonstrate superior image quality, robustness, and applicability to additional multi-modal scenarios compared with state-of-the-art methods.

Abstract

In photogrammetry, accurately fusing infrared (IR) and visible (VIS) spectra while preserving the geometric fidelity of visible features and incorporating thermal radiation is a significant challenge, particularly under extreme conditions. Existing methods often compromise visible imagery quality, impacting measurement accuracy. To solve this, we propose a region perception-based fusion framework that combines multi-exposure and multi-modal imaging using a spatially varying exposure (SVE) camera. This framework co-fuses multi-modal and multi-exposure data, overcoming single-exposure method limitations in extreme environments. The framework begins with region perception-based feature fusion to ensure precise multi-modal registration, followed by adaptive fusion with contrast enhancement. A structural similarity compensation mechanism, guided by regional saliency maps, optimizes IR-VIS spectral integration. Moreover, the framework adapts to single-exposure scenarios for robust fusion across different conditions. Experiments conducted on both synthetic and real-world data demonstrate superior image clarity and improved performance compared to state-of-the-art methods, as evidenced by both quantitative and visual evaluations.

Figures (15)

• Figure 1: Schematic illustration of different multi-modal image fusion tasks (first row: source images; second row: fusion results using a state-of-the-art method (LDFusion LDFusion); third row: our fusion results).
• Figure 2: Overall framework for multi-modal complementary fusion in extreme scenarios.
• Figure 3: Flowchart of the regional perception model construction module, where different colors in the regional perception map represent distinct regions.
• Figure 4: Features from images with different exposure levels are merged and filtered.
• Figure 5: Estimate the probability distribution of the optimal exposure in a given region.