Addressing the Depth-of-Field Constraint: A New Paradigm for High Resolution Multi-Focus Image Fusion

TL;DR

DOF constraints challenge all-in-focus imaging, and current MFIF methods suffer from data scarcity and domain gaps between synthetic and real data. The authors propose VAEEDOF, a two-stage system using a frozen 16-channel VAE distilled from StableDiffusion3 and a Fusion Module capable of fusing up to $7$ inputs, augmented by MattingMFIF, a 4K Blender-generated dataset for realistic training. The approach achieves state-of-the-art, artifact-free fused images and demonstrates strong generalization to real-world data with efficient inference, addressing both data scarcity and domain shift. This work advances high-resolution MFIF, with practical impact across photography, microscopy, and surveillance where reliable all-in-focus imagery is valuable, while acknowledging ethical considerations related to high-fidelity image synthesis.

Abstract

Multi-focus image fusion (MFIF) addresses the depth-of-field (DOF) limitations of optical lenses, where only objects within a specific range appear sharp. Although traditional and deep learning methods have advanced the field, challenges persist, including limited training data, domain gaps from synthetic datasets, and difficulties with regions lacking information. We propose VAEEDOF, a novel MFIF method that uses a distilled variational autoencoder for high-fidelity, efficient image reconstruction. Our fusion module processes up to seven images simultaneously, enabling robust fusion across diverse focus points. To address data scarcity, we introduce MattingMFIF, a new syntetic 4K dataset, simulating realistic DOF effects from real photographs. Our method achieves state-of-the-art results, generating seamless artifact-free fused images and bridging the gap between synthetic and real-world scenarios, offering a significant step forward in addressing complex MFIF challenges. The code, and weights are available here:

Figures (6)

• Figure 1: Overview of the architecture. VAE is pretrained and frozen during the training.
• Figure 2: The figure shows the differences between the original and the reconstructed image, with discrepancies highlighted in purple.
• Figure 3: Visualization of the patch combination process.
• Figure 4: The plot shows the LPSIS metric for different image patches categorized by their difficulty level. The size of each point represents the number of patches within a specific difficulty threshold, with higher thresholds corresponding to more challenging regions.
• Figure 5: The figure displays the output of all models under consideration. The first three rows are from our proposed dataset, while the remaining three are from the HighMF xiao2022dmdn dataset. The images are selected to provide a broad representation of each model’s performance.