Generative Multi-Focus Image Fusion

TL;DR

GMFF addresses the challenge of producing all-in-focus imagery from multi-focus stacks that may miss focal planes and exhibit edge artifacts. It decouples the task into a deterministic fusion stage using StackMFF V4 and a generative restoration stage with IFControlNet guided by latent diffusion priors, yielding state-of-the-art fusion quality and perceptual enhancement. While the generative stage improves detail and artifact suppression, it incurs higher computation and depends on accurate alignment, suggesting room for faster priors and geometry-aware enhancements. Overall, GMFF presents a flexible, two-stage framework that advances practical multi-focus fusion and opens avenues for incorporating stronger priors and faster diffusion backbones.

Abstract

Multi-focus image fusion aims to generate an all-in-focus image from a sequence of partially focused input images. Existing fusion algorithms generally assume that, for every spatial location in the scene, there is at least one input image in which that location is in focus. Furthermore, current fusion models often suffer from edge artifacts caused by uncertain focus estimation or hard-selection operations in complex real-world scenarios. To address these limitations, we propose a generative multi-focus image fusion framework, termed GMFF, which operates in two sequential stages. In the first stage, deterministic fusion is implemented using StackMFF V4, the latest version of the StackMFF series, and integrates the available focal plane information to produce an initial fused image. The second stage, generative restoration, is realized through IFControlNet, which leverages the generative capabilities of latent diffusion models to reconstruct content from missing focal planes, restore fine details, and eliminate edge artifacts. Each stage is independently developed and functions seamlessly in a cascaded manner. Extensive experiments demonstrate that GMFF achieves state-of-the-art fusion performance and exhibits significant potential for practical applications, particularly in scenarios involving complex multi-focal content. The implementation is publicly available at https://github.com/Xinzhe99/StackMFF-Series.

Figures (9)

• Figure 1: Comparison between prior models and our model (i.e., GMFF). (a) Deterministic models applicable to image stack fusion, represented mainly by the StackMFF series; (b) Representative methods that employ denoising probabilistic models for multi-focus image fusion, exemplified by FusionDiff li2024fusiondiff, which require pairwise iterative fusion to achieve image stack fusion; (c) The proposed GMFF framework employs a deterministic model for pre-fusion, while the denoising probabilistic model is used for image restoration rather than for fusion, as shown in (b).
• Figure 2: Overview of the proposed generative multi-focus image fusion framework (GMFF), which consists of two stages: deterministic fusion and generative restoration.
• Figure 3: Framework of the proposed StackMFF V4.
• Figure 4: Detailed transformer module comparison between StackMFF V3 (Pixel-wise Cross-Layer Attention, PCA) and StackMFF V4 (Spatial Aggregation Cross-Layer Attention, SACA).
• Figure 5: Comparison of fusion results produced by various methods on the Mobile Depth dataset suwajanakorn2015depth. The examples correspond to "keyboard" and "balls", respectively.