Searching a Compact Architecture for Robust Multi-Exposure Image Fusion

TL;DR

This work tackles robust multi-exposure fusion (MEF) under severe exposure gaps and device motion by introducing a hardware-aware neural architecture search framework. The key idea is a MEF-specific architecture built from a Self-alignment Module (SAM) and a Detail Repletion Module (DRM), guided by scene relighting to harmonize illumination and deformable alignment to reduce misalignment artifacts, with a NAS process that enforces latency constraints. The approach yields compact models that outperform or rival state-of-the-art methods in PSNR, SSIM, LPIPS, and FSIM, while achieving substantial reductions in parameter count and inference time. The results demonstrate strong generalization to misaligned and arbitrarily exposure-ratio scenarios, offering practical gains for real-time MEF on devices with limited compute resources.

Abstract

In recent years, learning-based methods have achieved significant advancements in multi-exposure image fusion. However, two major stumbling blocks hinder the development, including pixel misalignment and inefficient inference. Reliance on aligned image pairs in existing methods causes susceptibility to artifacts due to device motion. Additionally, existing techniques often rely on handcrafted architectures with huge network engineering, resulting in redundant parameters, adversely impacting inference efficiency and flexibility. To mitigate these limitations, this study introduces an architecture search-based paradigm incorporating self-alignment and detail repletion modules for robust multi-exposure image fusion. Specifically, targeting the extreme discrepancy of exposure, we propose the self-alignment module, leveraging scene relighting to constrain the illumination degree for following alignment and feature extraction. Detail repletion is proposed to enhance the texture details of scenes. Additionally, incorporating a hardware-sensitive constraint, we present the fusion-oriented architecture search to explore compact and efficient networks for fusion. The proposed method outperforms various competitive schemes, achieving a noteworthy 3.19\% improvement in PSNR for general scenarios and an impressive 23.5\% enhancement in misaligned scenarios. Moreover, it significantly reduces inference time by 69.1\%. The code will be available at https://github.com/LiuZhu-CV/CRMEF.

Figures (11)

• Figure 1: Visual comparison with representative MEF methods on general and misaligned fusion scenarios. The left figure plots the ranking with these competitors under general and misaligned scenarios. Other figures shows the obvious comparison with patch-based scheme MEFSPD li2021detail and learning-based method MEFGAN xu2020mef.
• Figure 2: Schematic diagram of the proposed architecture. The super-architecture for multi-exposure fusion consists of Self-Alignment Module (SAM) and Detail Repletion Module (DRM). Search spaces of SAM and for DRM are also illustrated respectively.
• Figure 3: Qualitative comparison with nine state-of-the-art methods. The signal maps provide the differences of pixel intensity with the ground truth.
• Figure 4: Qualitative comparison with nine state-of-the-art methods. The signal maps provide the differences of pixel intensity with ground truth.
• Figure 5: Qualitative comparison with nine state-of-the-art methods on the dataset ma2017robust without ground truth.