Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MagicFuse: Single Image Fusion for Visual and Semantic Reinforcement

Hao Zhang, Yanping Zha, Zizhuo Li, Meiqi Gong, Jiayi Ma

TL;DR

MagicFuse addresses cross-spectral scene understanding under harsh conditions using only a single degraded visible image. It introduces a diffusion-based framework with intra-spectral knowledge reinforcement (IKR) and cross-spectral knowledge generation (CKG), fused through a multi-domain knowledge fusion (MKF) module to yield a MagImg rich in visual and semantic cues. A segmentation head guides visual-semantic coupling and a radiation-map-based modulation preserves semantic objects during fusion. Extensive experiments show competitive or superior visual and semantic performance against multi-modal fusion benchmarks and demonstrate generalization to unseen data and even Cityscapes, highlighting practical impact in low-cost sensing scenarios.

Abstract

This paper focuses on a highly practical scenario: how to continue benefiting from the advantages of multi-modal image fusion under harsh conditions when only visible imaging sensors are available. To achieve this goal, we propose a novel concept of single-image fusion, which extends conventional data-level fusion to the knowledge level. Specifically, we develop MagicFuse, a novel single image fusion framework capable of deriving a comprehensive cross-spectral scene representation from a single low-quality visible image. MagicFuse first introduces an intra-spectral knowledge reinforcement branch and a cross-spectral knowledge generation branch based on the diffusion models. They mine scene information obscured in the visible spectrum and learn thermal radiation distribution patterns transferred to the infrared spectrum, respectively. Building on them, we design a multi-domain knowledge fusion branch that integrates the probabilistic noise from the diffusion streams of these two branches, from which a cross-spectral scene representation can be obtained through successive sampling. Then, we impose both visual and semantic constraints to ensure that this scene representation can satisfy human observation while supporting downstream semantic decision-making. Extensive experiments show that our MagicFuse achieves visual and semantic representation performance comparable to or even better than state-of-the-art fusion methods with multi-modal inputs, despite relying solely on a single degraded visible image.

MagicFuse: Single Image Fusion for Visual and Semantic Reinforcement

TL;DR

MagicFuse addresses cross-spectral scene understanding under harsh conditions using only a single degraded visible image. It introduces a diffusion-based framework with intra-spectral knowledge reinforcement (IKR) and cross-spectral knowledge generation (CKG), fused through a multi-domain knowledge fusion (MKF) module to yield a MagImg rich in visual and semantic cues. A segmentation head guides visual-semantic coupling and a radiation-map-based modulation preserves semantic objects during fusion. Extensive experiments show competitive or superior visual and semantic performance against multi-modal fusion benchmarks and demonstrate generalization to unseen data and even Cityscapes, highlighting practical impact in low-cost sensing scenarios.

Abstract

This paper focuses on a highly practical scenario: how to continue benefiting from the advantages of multi-modal image fusion under harsh conditions when only visible imaging sensors are available. To achieve this goal, we propose a novel concept of single-image fusion, which extends conventional data-level fusion to the knowledge level. Specifically, we develop MagicFuse, a novel single image fusion framework capable of deriving a comprehensive cross-spectral scene representation from a single low-quality visible image. MagicFuse first introduces an intra-spectral knowledge reinforcement branch and a cross-spectral knowledge generation branch based on the diffusion models. They mine scene information obscured in the visible spectrum and learn thermal radiation distribution patterns transferred to the infrared spectrum, respectively. Building on them, we design a multi-domain knowledge fusion branch that integrates the probabilistic noise from the diffusion streams of these two branches, from which a cross-spectral scene representation can be obtained through successive sampling. Then, we impose both visual and semantic constraints to ensure that this scene representation can satisfy human observation while supporting downstream semantic decision-making. Extensive experiments show that our MagicFuse achieves visual and semantic representation performance comparable to or even better than state-of-the-art fusion methods with multi-modal inputs, despite relying solely on a single degraded visible image.
Paper Structure (15 sections, 14 equations, 9 figures, 7 tables, 1 algorithm)

This paper contains 15 sections, 14 equations, 9 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Formulation
  4. Knowledge Reinforcement and Generation
  5. Knowledge Fusion
  6. Visual-Semantic Reinforcement
  7. Optimization Scheme
  8. Experiments
  9. Configuration
  10. Visual Representation Comparison
  11. Semantic Representation Comparison
  12. Generalization of Knowledge Fusion
  13. Ablation Studies
  14. Beyond Fusion Scenarios
  15. Conclusion

Figures (9)

  • Figure 1: Under limited sensing conditions, existing image fusion methods fail completely, whereas our MagicFuse still achieves promising cross-spectral scene representation.
  • Figure 2: Overall Framework of our proposed MagicFuse.
  • Figure 3: Qualitative visual representation comparison.
  • Figure 4: Qualitative semantic representation comparison.
  • Figure 5: Qualitative visual representation generalization.
  • ...and 4 more figures