Multi-Grained Text-Guided Image Fusion for Multi-Exposure and Multi-Focus Scenarios

TL;DR

This work tackles the challenge of fusing images captured under different exposures or focal depths by leveraging multi-grained textual guidance. MTIF generates detail-, structure-, and semantic-level textual descriptions via a large language model and aligns them with visual features through hierarchical cross-modal modulation, reinforced by a semantic-driven visual enrichment and a multi-grained loss. The approach yields superior MEF and MFF results across multiple datasets and metrics, with ablations confirming the effectiveness of each component. The framework demonstrates the value of integrating vision-language priors into low-level vision tasks and suggests broader applicability to cross-modal fusion tasks.

Abstract

Image fusion aims to synthesize a single high-quality image from a pair of inputs captured under challenging conditions, such as differing exposure levels or focal depths. A core challenge lies in effectively handling disparities in dynamic range and focus depth between the inputs. With the advent of vision-language models, recent methods incorporate textual descriptions as auxiliary guidance to enhance fusion quality. However, simply incorporating coarse-grained descriptions hampers the understanding of fine-grained details and poses challenges for precise cross-modal alignment. To address these limitations, we propose Multi-grained Text-guided Image Fusion (MTIF), a novel fusion paradigm with three key designs. First, it introduces multi-grained textual descriptions that separately capture fine details, structural cues, and semantic content, guiding image fusion through a hierarchical cross-modal modulation module. Second, it involves supervision signals at each granularity to facilitate alignment between visual and textual features and enhance the utility of auxiliary text. Third, it adopts a saliency-driven enrichment module to augment training data with dense semantic content, further strengthening the cross-modal modulation and alignment. Extensive experiments show that MTIF consistently outperforms previous methods on both multi-exposure and multi-focus image fusion tasks.

Figures (10)

• Figure 1: Unlike existing image fusion approaches that either solely rely on visual features as in (a) or simply introduce coarse-grained textual auxiliaries as in (b), the proposed multi-grained textual guidance framework introduces multi-grained textual information that captures fine details, structural cues, and semantic content to facilitate high-quality image fusion as shown in (c).
• Figure 2: Similarity analysis of multi-grained textual and visual features. Left: Similarity across textual descriptions at the detail, structure, and semantic levels. The relatively low scores imply that these descriptions are semantically complementary, providing richer information than a single-grained textual description. Right: Cross-modal similarity analysis between textual and visual features at multiple levels. The findings indicate that same-level similarities are higher than cross-level similarities, which inspires the multi-grained modulation design of MTIF.
• Figure 3: Architecture of the proposed MTIF, which synthesizes a single image from input multi-exposure or multi-focus pairs. First, we utilize a multi-modal large language model to generate multi-grained textual descriptions that capture detail-level, structure-level, and semantic-level information from the input images. Second, we modulate the multi-scale visual features with the corresponding textual features through a text-guided visual modulation module. Moreover, we introduce a multi-grained supervision strategy, which progressively bridges the cross-modal semantic gaps and enables effective feature modulation across all semantic levels. Additionally, we involve semantic-driven visual enrichment to enhance both the diversity and density of the training data.
• Figure 4: Pipeline of the Semantic-Driven Visual Enrichment module. First, a semantic saliency detector identifies semantically important regions of the input images. Second, a semantic-aware cropping strategy crops a central region with the highest saliency and four surrounding peripheral patches. This module increases the information density and diversity of the training data and facilitates the learning of multi-grained textual guidance.
• Figure 5: Visualization comparison of fusion results on the SICE cai2018learning dataset for the multi-exposure image fusion task.